2024-09-16T03:38:56.156 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:38:56.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:38:56.254 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920 branch: squid description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905920' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbV6wB7THjGtLpf0taXTtshOVbOHYhAjEIfLUS7sLcqzJMhonZcrEFTyUJCY3ETyirKkKVgIWj9c9IgQWhV9AY= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM4s4lawqQsXThQpAFIXzWhagoeL3dzAlhJPqCxANQiLVUH0Q/d8oW8kG9TULPdB2EABd4VLC+21YEx9cBMlTc= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:38:56.254 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:38:56.255 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:38:56.255 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:38:56.257 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:38:56.277 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:38:56.277 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:38:56.277 INFO:teuthology.packaging:ref: None 2024-09-16T03:38:56.277 INFO:teuthology.packaging:tag: None 2024-09-16T03:38:56.277 INFO:teuthology.packaging:branch: squid 2024-09-16T03:38:56.277 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:38:56.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T03:38:56.482 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T03:38:56.484 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:38:56.490 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:38:56.491 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:38:56.528 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:38:56.544 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:38:56.550 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:38:56.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:33:24.153759', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbV6wB7THjGtLpf0taXTtshOVbOHYhAjEIfLUS7sLcqzJMhonZcrEFTyUJCY3ETyirKkKVgIWj9c9IgQWhV9AY='} 2024-09-16T03:38:56.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:33:24.152063', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM4s4lawqQsXThQpAFIXzWhagoeL3dzAlhJPqCxANQiLVUH0Q/d8oW8kG9TULPdB2EABd4VLC+21YEx9cBMlTc='} 2024-09-16T03:38:56.589 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:38:56.595 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-16T03:38:56.595 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-16T03:38:56.595 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:38:56.652 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f14e1593ac0>, signals=[15]) 2024-09-16T03:38:56.653 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:38:56.658 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:38:56.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:38:56.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:38:56.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:38:56.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:38:56.817 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:38:56.823 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-09-16T03:38:56.842 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-09-16T03:38:56.842 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:38:56.902 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-09-16T03:38:56.921 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-09-16T03:38:56.939 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:38:56.996 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:38:56.997 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:38:56.997 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-09-16T03:38:57.021 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:38:57.030 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:38:57.037 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:38:57.037 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:38:57.039 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:38:57.056 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:38:57.062 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:38:57.062 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:38:57.097 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:38:57.124 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:38:57.180 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:38:57.195 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:38:57.195 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:38:57.195 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:38:57.196 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:38:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:38:57.212 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-16T03:38:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:38:57.213 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-09-16T03:38:57.269 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:38:57.270 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-503.el9.x86_64 2024-09-16T03:38:57.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-09-16T03:38:57.271 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:38:57.271 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-503.el9.x86_64 2024-09-16T03:38:57.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-09-16T03:39:01.904 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-09-16T03:39:02.050 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.2 MB 00:03 2024-09-16T03:39:03.760 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-16T03:39:03.805 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-16T03:39:07.985 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T03:39:09.460 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.5 MB 00:03 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T03:39:12.423 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-16T03:39:13.744 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-16T03:39:16.153 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-09-16T03:39:18.185 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:39:18.185 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:39:18.292 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:39:18.299 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:39:18.299 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:39:18.300 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 79 M 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 126 M 2024-09-16T03:39:18.301 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:39:18.738 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 77 kB/s | 23 kB 00:00 2024-09-16T03:39:19.241 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-16T03:39:21.206 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:39:21.206 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:39:21.295 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:39:21.304 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:39:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-09-16T03:39:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:39:21.306 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 79 M 2024-09-16T03:39:21.306 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 126 M 2024-09-16T03:39:21.306 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:39:21.675 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 193 kB/s | 23 kB 00:00 2024-09-16T03:39:22.359 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-16T03:39:22.967 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 22 MB/s | 29 MB 00:01 2024-09-16T03:39:23.285 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 20 MB/s | 35 MB 00:01 2024-09-16T03:39:23.285 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:39:23.285 INFO:teuthology.orchestra.run.smithi163.stdout:Total 40 MB/s | 79 MB 00:01 2024-09-16T03:39:23.966 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:39:24.337 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:39:24.338 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:39:25.312 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:39:25.313 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:39:27.766 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:39:28.472 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:39:28.569 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:39:31.378 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:39:31.503 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:39:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:14 2024-09-16T03:39:36.348 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:39:36.386 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:39:41.211 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:39:51.082 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 904 kB/s | 29 MB 00:32 2024-09-16T03:40:00.756 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 836 kB/s | 35 MB 00:42 2024-09-16T03:40:00.756 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:40:00.756 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.9 MB/s | 79 MB 00:42 2024-09-16T03:40:01.439 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:40:01.805 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:40:01.805 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:40:02.824 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:40:02.825 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:40:05.290 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:40:05.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:40:06.072 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:40:08.850 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:40:08.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:40:13.801 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:40:13.909 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:18.795 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:19.549 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:19.906 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:19.907 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:40:19.907 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:40:19.907 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:40:20.790 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:20.790 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:40:20.790 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:40:20.790 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:20.791 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:40:20.791 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:40:20.791 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:40:20.791 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:40:20.791 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:40:21.166 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:40:21.256 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:21.256 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:40:21.256 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:40:21.256 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:40:21.256 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:40:21.257 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:40:21.257 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:40:21.257 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:40:21.315 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:21.316 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:40:21.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 82.el9 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8274670 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:40:21.348 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-09-16T03:40:21.349 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:40:21.350 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:40:21.350 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:40:21.350 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:40:21.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:40:21.964 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-09-16T03:40:23.262 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:40:23.290 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-09-16T03:40:23.292 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-09-16T03:40:23.308 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.8IgYjXmwza 2024-09-16T03:40:23.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8IgYjXmwza 2024-09-16T03:40:23.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.8IgYjXmwza 2024-09-16T03:40:23.530 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.8IgYjXmwza is 6KB 2024-09-16T03:40:23.579 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.8IgYjXmwza 2024-09-16T03:40:23.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:40:23.657 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:40:23.657 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:40:23.657 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:40:23.657 INFO:teuthology.orchestra.run.smithi163.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:40:23.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:40:23.809 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-09-16T03:40:53.812 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:40:53.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:40:53.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:40:56.771 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:56.811 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:57.172 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:57.172 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:40:57.172 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:40:57.172 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:40:58.008 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:40:58.370 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:40:58.465 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.466 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:40:58.466 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:40:58.466 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:40:58.466 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:40:58.466 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:40:58.466 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:40:58.466 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:40:58.525 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:40:58.525 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:40:58.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 82.el9 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-09-16T03:40:58.565 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8274670 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:40:58.566 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:40:58.567 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:40:58.567 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-09-16T03:40:58.567 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:40:58.567 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:40:58.568 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:40:58.568 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:40:58.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:40:59.201 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-09-16T03:41:00.486 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:41:00.514 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-09-16T03:41:00.516 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-16T03:41:00.531 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.G97sUib8Y9 2024-09-16T03:41:01.445 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.G97sUib8Y9 2024-09-16T03:41:01.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.G97sUib8Y9 2024-09-16T03:41:01.636 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.G97sUib8Y9 is 6KB 2024-09-16T03:41:01.657 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.G97sUib8Y9 2024-09-16T03:41:01.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:41:01.748 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:41:01.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:41:01.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-09-16T03:41:12.385 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T03:41:21.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:41:21.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:41:24.449 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T03:41:31.913 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:41:31.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-16T03:41:31.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:41:36.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:41:36.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:41:39.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T03:41:54.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:41:54.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:42:13.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-09-16T03:42:31.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:42:31.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:42:31.446 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-09-16T03:42:31.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:42:31.861 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:42:31.861 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-09-16T03:42:31.876 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:42:31.876 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:42:31.876 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:42:31.877 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:42:31.916 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:42:32.878 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:42:32.878 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:42:32.942 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-09-16T03:42:32.972 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:40.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-16T03:42:40.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:42:41.506 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-16T03:42:41.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-16T03:42:41.887 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:42:41.888 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-16T03:42:41.903 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:42:41.903 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:42:41.903 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:42:41.903 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:42:42.904 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:42:42.904 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:42:42.967 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-09-16T03:42:42.991 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:42.991 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:42:42.998 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:42:42.998 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:42:43.002 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:42:43.020 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:42:43.047 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:42:43.057 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:42:43.058 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:43.060 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:43.273 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:42:43.280 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:42:43.281 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:42:43.283 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:42:43.315 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:42:43.319 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:42:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:42:43.331 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:42:43.332 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:42:43.352 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:42:43.352 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:42:43.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:42:43.400 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:42:43.409 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:42:43.409 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:42:43.435 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:42:43.457 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:42:43.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:42:43.578 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:42:43.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:42:43.699 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:42:43.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:42:43.765 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:42:43.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:42:43.875 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:42:43.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:42:43.961 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:42:43.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:42:44.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-09-16T03:42:44.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-09-16T03:42:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:44.119 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:44.209 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:42:44.218 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:42:44.218 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:42:44.241 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:42:44.248 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T03:42:44.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-09-16T03:42:44.325 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-09-16T03:42:44.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-09-16T03:42:44.374 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-09-16T03:42:44.376 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:42:44.376 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-09-16T03:42:44.405 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-09-16T03:42:44.406 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-09-16T03:42:44.449 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-09-16T03:42:44.449 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-09-16T03:42:44.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:44.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:42:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:44.518 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:42:44.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:42:44.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:42:44.592 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:42:44.601 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:42:44.612 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:42:44.613 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:46:30.988 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-09-16T03:46:30.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-16T03:46:30.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:46:31.080 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-16T03:46:31.168 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-16T03:46:31.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:46:31.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:46:31.245 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-09-16T03:46:31.324 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-09-16T03:46:31.324 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:46:31.334 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:46:31.334 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:46:31.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:46:31.337 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:46:31.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:46:31.365 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:46:31.380 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:46:31.393 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:46:31.404 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-09-16T03:46:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:46:31.415 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-09-16T03:46:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:46:31.431 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-09-16T03:46:31.442 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-09-16T03:46:31.444 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:46:31.457 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:46:31.471 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:46:31.501 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.504 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.526 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.530 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:46:31.531 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:46:31.539 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:46:31.539 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T03:46:31.539 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T03:46:31.539 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:46:31.546 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:46:31.546 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:46:31.546 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:46:31.547 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:46:31.549 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T03:46:31.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:46:31.694 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:46:31.694 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:46:31.697 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T03:46:31.697 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T03:46:31.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:46:31.806 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:46:31.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:46:31.813 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:46:31.813 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:46:31.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:46:31.832 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:46:31.832 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:46:31.839 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:46:31.839 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:46:31.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:46:31.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:46:31.973 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:46:31.978 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:46:32.037 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-09-16T03:46:32.039 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-09-16T03:46:32.042 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-09-16T03:46:32.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-09-16T03:46:32.268 INFO:teuthology.orchestra.run.smithi070.stdout:45 files removed 2024-09-16T03:46:32.273 INFO:teuthology.orchestra.run.smithi163.stdout:45 files removed 2024-09-16T03:46:32.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2024-09-16T03:46:32.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-09-16T03:46:32.849 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 315 kB/s | 83 kB 00:00 2024-09-16T03:46:32.856 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 304 kB/s | 83 kB 00:00 2024-09-16T03:46:33.134 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T03:46:33.140 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T03:46:33.407 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-16T03:46:33.409 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T03:46:33.882 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-16T03:46:35.109 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T03:46:36.928 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-16T03:46:37.283 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 20 MB 00:02 2024-09-16T03:46:41.608 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-16T03:46:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-16T03:46:43.209 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-16T03:46:43.979 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 19 kB 00:00 2024-09-16T03:46:44.913 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-16T03:46:45.142 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-16T03:46:50.327 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T03:46:50.639 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:46:52.728 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:46:52.748 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:46:52.749 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:46:52.750 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:46:52.751 INFO:teuthology.orchestra.run.smithi070.stdout:Install 30 Packages 2024-09-16T03:46:52.752 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2024-09-16T03:46:52.752 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:46:52.752 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 60 M 2024-09-16T03:46:52.752 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:46:53.017 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T03:46:53.028 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:46:53.029 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:46:53.030 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout:Install 30 Packages 2024-09-16T03:46:53.031 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-09-16T03:46:53.032 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:46:53.032 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 60 M 2024-09-16T03:46:53.032 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:46:53.394 INFO:teuthology.orchestra.run.smithi070.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-16T03:46:53.477 INFO:teuthology.orchestra.run.smithi070.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 302 kB/s | 25 kB 00:00 2024-09-16T03:46:53.586 INFO:teuthology.orchestra.run.smithi070.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.8 MB/s | 751 kB 00:00 2024-09-16T03:46:53.695 INFO:teuthology.orchestra.run.smithi070.stdout:(4/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 11 MB 00:00 2024-09-16T03:46:53.720 INFO:teuthology.orchestra.run.smithi070.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 3.7 MB/s | 506 kB 00:00 2024-09-16T03:46:53.820 INFO:teuthology.orchestra.run.smithi070.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 451 kB/s | 45 kB 00:00 2024-09-16T03:46:54.029 INFO:teuthology.orchestra.run.smithi070.stdout:(7/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 16 MB/s | 5.3 MB 00:00 2024-09-16T03:46:54.095 INFO:teuthology.orchestra.run.smithi070.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.4 MB/s | 161 kB 00:00 2024-09-16T03:46:54.120 INFO:teuthology.orchestra.run.smithi070.stdout:(9/32): python3-ceph-common-19.1.1-326.g8921c8e 453 kB/s | 136 kB 00:00 2024-09-16T03:46:54.180 INFO:teuthology.orchestra.run.smithi070.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 3.7 MB/s | 320 kB 00:00 2024-09-16T03:46:54.213 INFO:teuthology.orchestra.run.smithi070.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 3.2 MB/s | 300 kB 00:00 2024-09-16T03:46:54.421 INFO:teuthology.orchestra.run.smithi070.stdout:(12/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 14 MB/s | 21 MB 00:01 2024-09-16T03:46:54.447 INFO:teuthology.orchestra.run.smithi070.stdout:(13/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 372 kB/s | 99 kB 00:00 2024-09-16T03:46:54.557 INFO:teuthology.orchestra.run.smithi070.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 310 kB/s | 107 kB 00:00 2024-09-16T03:46:54.574 INFO:teuthology.orchestra.run.smithi070.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 355 kB/s | 45 kB 00:00 2024-09-16T03:46:54.600 INFO:teuthology.orchestra.run.smithi070.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 902 kB/s | 160 kB 00:00 2024-09-16T03:46:54.650 INFO:teuthology.orchestra.run.smithi070.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-16T03:46:54.684 INFO:teuthology.orchestra.run.smithi070.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-16T03:46:54.690 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 7.7 MB/s | 11 MB 00:01 2024-09-16T03:46:54.717 INFO:teuthology.orchestra.run.smithi070.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-16T03:46:54.743 INFO:teuthology.orchestra.run.smithi070.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-16T03:46:54.768 INFO:teuthology.orchestra.run.smithi070.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-16T03:46:54.793 INFO:teuthology.orchestra.run.smithi070.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-16T03:46:54.852 INFO:teuthology.orchestra.run.smithi070.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 601 kB/s | 26 kB 00:00 2024-09-16T03:46:54.885 INFO:teuthology.orchestra.run.smithi070.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-16T03:46:54.899 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 13 MB/s | 21 MB 00:01 2024-09-16T03:46:54.911 INFO:teuthology.orchestra.run.smithi070.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-09-16T03:46:54.916 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 112 kB/s | 25 kB 00:00 2024-09-16T03:46:54.936 INFO:teuthology.orchestra.run.smithi070.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:46:54.966 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 751 kB 00:00 2024-09-16T03:46:54.977 INFO:teuthology.orchestra.run.smithi070.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T03:46:54.992 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): libradosstriper1-19.1.1-326.g8921c8ef.e 6.6 MB/s | 506 kB 00:00 2024-09-16T03:46:55.034 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 1.1 MB/s | 45 kB 00:00 2024-09-16T03:46:55.036 INFO:teuthology.orchestra.run.smithi070.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-16T03:46:55.084 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 2.7 MB/s | 136 kB 00:00 2024-09-16T03:46:55.134 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 3.2 MB/s | 161 kB 00:00 2024-09-16T03:46:55.153 INFO:teuthology.orchestra.run.smithi070.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-16T03:46:55.218 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_64 21 MB/s | 5.3 MB 00:00 2024-09-16T03:46:55.313 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86 2.7 MB/s | 5.4 MB 00:01 2024-09-16T03:46:55.352 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-rados-19.1.1-326.g8921c8ef.el9 1.4 MB/s | 320 kB 00:00 2024-09-16T03:46:55.358 INFO:teuthology.orchestra.run.smithi070.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-09-16T03:46:55.377 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.5 MB/s | 99 kB 00:00 2024-09-16T03:46:55.403 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 1.6 MB/s | 300 kB 00:00 2024-09-16T03:46:55.429 INFO:teuthology.orchestra.run.smithi070.stdout:(31/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-16T03:46:55.520 INFO:teuthology.orchestra.run.smithi070.stdout:(32/32): librados2-19.1.1-326.g8921c8ef.el9.x86 7.1 MB/s | 3.4 MB 00:00 2024-09-16T03:46:55.526 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:46:55.526 INFO:teuthology.orchestra.run.smithi070.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-16T03:46:55.536 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-16T03:46:55.570 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 833 kB/s | 160 kB 00:00 2024-09-16T03:46:55.603 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): boost-program-options-1.75.0-8.el9.x86 426 kB/s | 107 kB 00:00 2024-09-16T03:46:55.630 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:46:55.668 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:46:55.668 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:46:55.679 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 564 kB/s | 42 kB 00:00 2024-09-16T03:46:55.712 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-16T03:46:55.738 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-16T03:46:55.763 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T03:46:55.789 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 474 kB/s | 36 kB 00:00 2024-09-16T03:46:55.814 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-16T03:46:55.873 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T03:46:55.898 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 383 kB/s | 26 kB 00:00 2024-09-16T03:46:55.923 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-09-16T03:46:55.949 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:46:56.017 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.7 MB/s | 839 kB 00:00 2024-09-16T03:46:56.049 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2024-09-16T03:46:56.125 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T03:46:56.266 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-09-16T03:46:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 15 MB/s | 3.2 MB 00:00 2024-09-16T03:46:56.366 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:46:56.367 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:46:56.742 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): librados2-19.1.1-326.g8921c8ef.el9.x86 4.9 MB/s | 3.4 MB 00:00 2024-09-16T03:46:56.748 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:46:56.748 INFO:teuthology.orchestra.run.smithi163.stdout:Total 16 MB/s | 60 MB 00:03 2024-09-16T03:46:56.858 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:46:56.895 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:46:56.895 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:46:57.571 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:46:57.571 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:46:57.852 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:46:57.937 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:46:57.967 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:46:57.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:46:58.015 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:46:58.043 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:46:58.226 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:46:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:46:58.641 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:46:58.761 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:46:58.888 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:46:58.983 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:46:59.012 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:46:59.050 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:46:59.075 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:46:59.112 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:46:59.132 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T03:46:59.155 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:46:59.180 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T03:46:59.206 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T03:46:59.224 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:46:59.228 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:46:59.248 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:46:59.255 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T03:46:59.296 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:46:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:46:59.398 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:46:59.440 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:46:59.452 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T03:46:59.482 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:46:59.522 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:46:59.563 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:46:59.592 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:46:59.641 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:46:59.859 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T03:46:59.908 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:46:59.955 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:46:59.996 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:47:00.000 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:47:00.085 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:47:00.095 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T03:47:00.120 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:47:00.134 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:47:00.161 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:47:00.197 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T03:47:00.225 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T03:47:00.273 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T03:47:00.347 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:47:00.380 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:47:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:47:00.460 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T03:47:00.577 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T03:47:00.612 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T03:47:00.642 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:47:00.652 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T03:47:00.673 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:47:00.685 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T03:47:00.714 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T03:47:00.728 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:47:00.748 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T03:47:00.797 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:47:00.826 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:47:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T03:47:01.119 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:47:01.220 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T03:47:01.249 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T03:47:01.302 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T03:47:01.814 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T03:47:01.847 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:47:01.918 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T03:47:01.998 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T03:47:02.033 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T03:47:03.112 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:03.147 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:03.711 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:03.750 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:47:03.835 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:47:03.835 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:47:03.835 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:47:03.835 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:03.885 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:04.372 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:04.422 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:04.981 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T03:47:05.035 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:47:05.124 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T03:47:05.124 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:47:05.124 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:47:05.124 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:13.997 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:13.997 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-09-16T03:47:13.997 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-09-16T03:47:13.997 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-09-16T03:47:13.998 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-09-16T03:47:13.998 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-09-16T03:47:13.998 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-09-16T03:47:13.998 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-09-16T03:47:13.998 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:14.988 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:47:15.014 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:47:15.014 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:15.014 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:47:15.014 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:47:15.014 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:47:15.015 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:15.016 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:47:15.108 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:47:15.108 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:15.167 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-09-16T03:47:15.168 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:16.169 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:47:16.195 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T03:47:16.196 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:16.196 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:47:16.196 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:47:16.196 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:47:16.196 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:16.197 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:47:16.281 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:47:16.282 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:47:16.283 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:47:16.284 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:47:16.284 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:47:16.284 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:47:16.284 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:47:16.284 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:47:16.285 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T03:47:16.285 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:16.285 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:47:16.286 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:47:16.286 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:47:16.286 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:16.286 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:47:16.286 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.442 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:47:17.443 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:17.444 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:17.445 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T03:47:17.576 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T03:47:17.577 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T03:47:17.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T03:47:17.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T03:47:17.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T03:47:17.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T03:47:17.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2024-09-16T03:47:18.314 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:18.446 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:47:18.447 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:18.448 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 49 M 2024-09-16T03:47:18.449 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 209 M 2024-09-16T03:47:18.449 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:18.674 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.675 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:47:18.676 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:47:18.677 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:18.678 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:18.763 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 758 kB/s | 163 kB 00:00 2024-09-16T03:47:18.970 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-09-16T03:47:19.105 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-09-16T03:47:19.130 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 522 kB/s | 304 kB 00:00 2024-09-16T03:47:19.536 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:19.679 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:19.679 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:47:19.679 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:19.679 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-09-16T03:47:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:19.681 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 49 M 2024-09-16T03:47:19.681 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 209 M 2024-09-16T03:47:19.681 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:19.945 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 751 kB/s | 163 kB 00:00 2024-09-16T03:47:20.056 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-16T03:47:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-16T03:47:20.086 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:20.106 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:20.106 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:20.503 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:20.503 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:20.504 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 392 kB/s | 304 kB 00:00 2024-09-16T03:47:20.588 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 100 kB/s | 64 kB 00:00 2024-09-16T03:47:21.044 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:47:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:47:21.224 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:47:21.313 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 31 MB/s | 49 MB 00:01 2024-09-16T03:47:21.315 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:21.316 INFO:teuthology.orchestra.run.smithi163.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-16T03:47:21.344 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:47:21.364 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:47:21.365 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:47:21.765 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:47:21.766 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:47:22.382 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:47:22.428 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:47:22.457 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:47:25.850 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:47:25.948 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:47:27.071 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T03:47:27.155 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:47:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:47:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:47:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:47:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:47:28.625 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:47:28.625 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:28.625 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:28.625 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:28.625 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:28.626 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:47:28.626 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:47:28.626 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:28.626 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:28.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2024-09-16T03:47:29.489 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:29.582 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T03:47:29.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T03:47:29.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T03:47:29.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:47:29.634 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:29.636 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:47:29.637 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:47:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:47:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout:Install 37 Packages 2024-09-16T03:47:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:29.641 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 30 M 2024-09-16T03:47:29.641 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 108 M 2024-09-16T03:47:29.641 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:29.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:47:29.905 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:29.906 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:30.218 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-09-16T03:47:30.423 INFO:teuthology.orchestra.run.smithi070.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 46 kB/s | 6.5 kB 00:00 2024-09-16T03:47:30.582 INFO:teuthology.orchestra.run.smithi070.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-16T03:47:30.649 INFO:teuthology.orchestra.run.smithi070.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.5 MB/s | 2.4 MB 00:00 2024-09-16T03:47:30.749 INFO:teuthology.orchestra.run.smithi070.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.4 MB/s | 249 kB 00:00 2024-09-16T03:47:30.816 INFO:teuthology.orchestra.run.smithi070.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-16T03:47:30.830 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:30.925 INFO:teuthology.orchestra.run.smithi070.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-16T03:47:30.967 INFO:teuthology.orchestra.run.smithi070.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T03:47:30.974 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:30.976 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T03:47:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:47:30.978 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:47:30.979 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:47:30.980 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:30.981 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:30.981 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:30.981 INFO:teuthology.orchestra.run.smithi163.stdout:Install 37 Packages 2024-09-16T03:47:30.981 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:30.981 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 30 M 2024-09-16T03:47:30.982 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 108 M 2024-09-16T03:47:30.982 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:31.009 INFO:teuthology.orchestra.run.smithi070.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T03:47:31.176 INFO:teuthology.orchestra.run.smithi070.stdout:(9/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-16T03:47:31.201 INFO:teuthology.orchestra.run.smithi070.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-09-16T03:47:31.234 INFO:teuthology.orchestra.run.smithi070.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-16T03:47:31.318 INFO:teuthology.orchestra.run.smithi070.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-16T03:47:31.499 INFO:teuthology.orchestra.run.smithi163.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 47 kB/s | 6.5 kB 00:00 2024-09-16T03:47:31.666 INFO:teuthology.orchestra.run.smithi163.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-16T03:47:31.717 INFO:teuthology.orchestra.run.smithi163.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.7 MB/s | 2.4 MB 00:00 2024-09-16T03:47:31.809 INFO:teuthology.orchestra.run.smithi163.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.7 MB/s | 249 kB 00:00 2024-09-16T03:47:31.868 INFO:teuthology.orchestra.run.smithi070.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-09-16T03:47:31.969 INFO:teuthology.orchestra.run.smithi070.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 235 kB/s | 172 kB 00:00 2024-09-16T03:47:31.994 INFO:teuthology.orchestra.run.smithi070.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 238 kB/s | 188 kB 00:00 2024-09-16T03:47:32.009 INFO:teuthology.orchestra.run.smithi163.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T03:47:32.109 INFO:teuthology.orchestra.run.smithi163.stdout:(6/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 7.7 MB/s | 4.7 MB 00:00 2024-09-16T03:47:32.301 INFO:teuthology.orchestra.run.smithi163.stdout:(7/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-16T03:47:32.335 INFO:teuthology.orchestra.run.smithi163.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.8 MB/s | 1.2 MB 00:00 2024-09-16T03:47:32.385 INFO:teuthology.orchestra.run.smithi163.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-16T03:47:32.444 INFO:teuthology.orchestra.run.smithi163.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T03:47:32.469 INFO:teuthology.orchestra.run.smithi163.stdout:(11/37): python3-ply-3.11-14.el9.noarch.rpm 296 kB/s | 106 kB 00:00 2024-09-16T03:47:32.494 INFO:teuthology.orchestra.run.smithi070.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 123 kB/s | 77 kB 00:00 2024-09-16T03:47:32.503 INFO:teuthology.orchestra.run.smithi163.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 670 kB/s | 135 kB 00:00 2024-09-16T03:47:32.529 INFO:teuthology.orchestra.run.smithi163.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-09-16T03:47:32.553 INFO:teuthology.orchestra.run.smithi070.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 74 kB/s | 42 kB 00:00 2024-09-16T03:47:32.562 INFO:teuthology.orchestra.run.smithi163.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-16T03:47:32.586 INFO:teuthology.orchestra.run.smithi070.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 83 kB/s | 51 kB 00:00 2024-09-16T03:47:32.587 INFO:teuthology.orchestra.run.smithi163.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 414 kB/s | 35 kB 00:00 2024-09-16T03:47:32.611 INFO:teuthology.orchestra.run.smithi070.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-16T03:47:32.613 INFO:teuthology.orchestra.run.smithi163.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 918 kB/s | 77 kB 00:00 2024-09-16T03:47:32.638 INFO:teuthology.orchestra.run.smithi163.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-16T03:47:32.653 INFO:teuthology.orchestra.run.smithi070.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-16T03:47:32.663 INFO:teuthology.orchestra.run.smithi163.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-16T03:47:32.679 INFO:teuthology.orchestra.run.smithi070.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 480 kB/s | 43 kB 00:00 2024-09-16T03:47:32.704 INFO:teuthology.orchestra.run.smithi070.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 434 kB/s | 11 kB 00:00 2024-09-16T03:47:32.714 INFO:teuthology.orchestra.run.smithi163.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-09-16T03:47:32.730 INFO:teuthology.orchestra.run.smithi070.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-16T03:47:32.739 INFO:teuthology.orchestra.run.smithi163.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-16T03:47:32.758 INFO:teuthology.orchestra.run.smithi070.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.4 MB/s | 358 kB 00:00 2024-09-16T03:47:32.773 INFO:teuthology.orchestra.run.smithi163.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-16T03:47:32.783 INFO:teuthology.orchestra.run.smithi070.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 224 kB/s | 18 kB 00:00 2024-09-16T03:47:32.789 INFO:teuthology.orchestra.run.smithi163.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 648 kB/s | 11 kB 00:00 2024-09-16T03:47:32.809 INFO:teuthology.orchestra.run.smithi070.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-09-16T03:47:32.815 INFO:teuthology.orchestra.run.smithi163.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-16T03:47:32.834 INFO:teuthology.orchestra.run.smithi070.stdout:(27/37): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-09-16T03:47:32.840 INFO:teuthology.orchestra.run.smithi163.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-16T03:47:32.857 INFO:teuthology.orchestra.run.smithi163.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 552 kB/s | 23 kB 00:00 2024-09-16T03:47:32.859 INFO:teuthology.orchestra.run.smithi070.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-16T03:47:32.884 INFO:teuthology.orchestra.run.smithi070.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-16T03:47:32.886 INFO:teuthology.orchestra.run.smithi163.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 676 kB/s | 19 kB 00:00 2024-09-16T03:47:32.910 INFO:teuthology.orchestra.run.smithi070.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T03:47:32.911 INFO:teuthology.orchestra.run.smithi163.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 365 kB/s | 20 kB 00:00 2024-09-16T03:47:32.935 INFO:teuthology.orchestra.run.smithi070.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T03:47:32.944 INFO:teuthology.orchestra.run.smithi163.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.7 MB/s | 46 kB 00:00 2024-09-16T03:47:32.960 INFO:teuthology.orchestra.run.smithi070.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T03:47:32.980 INFO:teuthology.orchestra.run.smithi163.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-16T03:47:32.985 INFO:teuthology.orchestra.run.smithi070.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-16T03:47:33.004 INFO:teuthology.orchestra.run.smithi163.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-09-16T03:47:33.011 INFO:teuthology.orchestra.run.smithi070.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T03:47:33.036 INFO:teuthology.orchestra.run.smithi070.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-16T03:47:33.046 INFO:teuthology.orchestra.run.smithi163.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T03:47:33.053 INFO:teuthology.orchestra.run.smithi070.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 478 kB/s | 20 kB 00:00 2024-09-16T03:47:33.071 INFO:teuthology.orchestra.run.smithi163.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-16T03:47:33.078 INFO:teuthology.orchestra.run.smithi070.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-16T03:47:33.083 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:33.083 INFO:teuthology.orchestra.run.smithi070.stdout:Total 8.8 MB/s | 30 MB 00:03 2024-09-16T03:47:33.096 INFO:teuthology.orchestra.run.smithi163.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-16T03:47:33.130 INFO:teuthology.orchestra.run.smithi163.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-16T03:47:33.155 INFO:teuthology.orchestra.run.smithi163.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-16T03:47:33.189 INFO:teuthology.orchestra.run.smithi163.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-16T03:47:33.236 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:33.276 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:33.277 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:33.280 INFO:teuthology.orchestra.run.smithi163.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-09-16T03:47:33.284 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:33.284 INFO:teuthology.orchestra.run.smithi163.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-16T03:47:33.427 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:47:33.467 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:47:33.467 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:47:33.752 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:33.752 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:33.950 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:47:33.951 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:47:35.152 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:35.214 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:47:35.252 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:47:35.284 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:47:35.349 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:47:35.355 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:47:35.399 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:47:35.425 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:47:35.432 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:47:35.469 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:47:35.503 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:47:35.524 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:47:35.574 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:47:35.594 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:47:35.635 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:47:35.636 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:47:35.669 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:47:35.670 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:47:35.700 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:47:35.735 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:47:35.760 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:47:35.768 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:47:35.808 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:47:35.840 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:47:35.844 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:47:35.878 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:47:35.880 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:47:35.915 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:47:35.926 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:47:35.953 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:47:35.976 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:47:35.996 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:47:36.030 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:47:36.049 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:47:36.070 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:47:36.105 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:47:36.140 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:47:36.188 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:47:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:47:36.220 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:47:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:47:36.305 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:47:36.339 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:47:36.378 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:47:36.427 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:47:36.456 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:47:36.482 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:47:36.536 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:47:36.587 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:47:36.626 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:47:36.668 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:47:36.684 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:47:36.724 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:47:36.748 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:47:36.805 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:47:36.890 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:47:36.932 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:47:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:47:36.989 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:47:37.066 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:47:37.084 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:47:37.172 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:47:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:47:37.233 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:47:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:47:37.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:37.349 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:47:37.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T03:47:37.535 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:47:37.561 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:38.718 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:47:38.745 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:38.994 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:47:39.021 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T03:47:39.021 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:39.021 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:47:39.021 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:47:39.021 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:47:39.022 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:39.185 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:47:39.209 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:47:39.209 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:39.209 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:47:39.209 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:47:39.210 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:47:39.210 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:39.435 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:47:39.460 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:39.461 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:47:39.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:39.523 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:47:39.712 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:47:39.736 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:39.800 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:47:41.233 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:47:41.234 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:47:41.235 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:47:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:47:41.237 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T03:47:41.645 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:47:41.646 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:47:41.647 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:47:41.648 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:47:41.648 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:47:41.648 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.205 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:47:42.206 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:47:42.207 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:42.208 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:42.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2024-09-16T03:47:42.624 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:47:42.625 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:47:42.626 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:47:42.627 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:42.628 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:43.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-09-16T03:47:43.166 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:43.255 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:47:43.296 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:43.297 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:47:43.297 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:43.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2024-09-16T03:47:43.591 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:43.682 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:47:43.725 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:43.726 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:47:43.726 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:43.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-09-16T03:47:43.921 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:44.052 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:44.053 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:44.053 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:47:44.053 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:44.053 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:44.053 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 765 k 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 771 k 2024-09-16T03:47:44.054 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:44.337 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-16T03:47:44.337 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:44.337 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-16T03:47:44.337 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:44.343 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:44.343 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:44.349 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:44.349 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:44.373 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:44.426 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:44.507 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:44.507 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:44.507 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 765 k 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 771 k 2024-09-16T03:47:44.508 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:44.764 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-16T03:47:44.764 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:44.765 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-16T03:47:44.765 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:47:44.770 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:47:44.770 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:47:44.777 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:47:44.777 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:47:44.849 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:47:44.914 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.007 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.300 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.393 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.587 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.587 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:45.587 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:45.588 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:45.588 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:45.588 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:45.721 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:47:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:47:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:45.972 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:45.972 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:46.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:47:46.368 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:46.504 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:47:46.505 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:46.506 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 145 k 2024-09-16T03:47:46.506 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 435 k 2024-09-16T03:47:46.506 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:46.694 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 770 kB/s | 145 kB 00:00 2024-09-16T03:47:46.695 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:46.695 INFO:teuthology.orchestra.run.smithi070.stdout:Total 764 kB/s | 145 kB 00:00 2024-09-16T03:47:46.695 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:46.703 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:46.703 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:46.760 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:46.761 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:46.775 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:46.908 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:46.909 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:46.909 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T03:47:46.909 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:46.909 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 145 k 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 435 k 2024-09-16T03:47:46.910 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:46.921 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:47.004 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:47.029 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:47.029 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:47.029 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:47:47.029 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:47.122 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 685 kB/s | 145 kB 00:00 2024-09-16T03:47:47.123 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:47.123 INFO:teuthology.orchestra.run.smithi163.stdout:Total 680 kB/s | 145 kB 00:00 2024-09-16T03:47:47.123 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:47:47.131 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:47:47.131 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:47:47.188 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:47:47.188 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:47:47.339 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:47:47.432 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:47.457 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:47.458 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:47:47.458 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:47:47.458 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:47.947 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:48.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:48.380 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:48.625 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-09-16T03:47:48.737 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:48.829 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:47:48.872 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:48.872 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:47:48.873 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:48.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:47:49.202 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:49.295 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:49.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:47:49.495 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:49.627 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:49.628 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:49.628 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:47:49.628 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:47:49.629 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 8.7 M 2024-09-16T03:47:49.630 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 104 M 2024-09-16T03:47:49.631 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:49.949 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): ceph-prometheus-alerts-19.1.1-326.g8921 111 kB/s | 16 kB 00:00 2024-09-16T03:47:49.974 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): ceph-grafana-dashboards-19.1.1-326.g892 184 kB/s | 32 kB 00:00 2024-09-16T03:47:49.997 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-16T03:47:50.130 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:47:50.132 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-09-16T03:47:50.133 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:50.134 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 8.7 M 2024-09-16T03:47:50.134 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 104 M 2024-09-16T03:47:50.134 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:50.158 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 10 MB/s | 3.6 MB 00:00 2024-09-16T03:47:50.217 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-16T03:47:50.292 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-09-16T03:47:50.318 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 193 kB/s | 19 kB 00:00 2024-09-16T03:47:50.351 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.1 MB/s | 31 kB 00:00 2024-09-16T03:47:50.385 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.5 MB/s | 144 kB 00:00 2024-09-16T03:47:50.410 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-16T03:47:50.493 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.4 MB/s | 2.0 MB 00:00 2024-09-16T03:47:50.498 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:50.498 INFO:teuthology.orchestra.run.smithi070.stdout:Total 10 MB/s | 8.7 MB 00:00 2024-09-16T03:47:50.546 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:50.563 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:50.563 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:50.619 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 205 kB/s | 32 kB 00:00 2024-09-16T03:47:50.644 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 91 kB/s | 16 kB 00:00 2024-09-16T03:47:50.774 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:50.775 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:50.836 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-16T03:47:50.895 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 8.3 MB/s | 3.6 MB 00:00 2024-09-16T03:47:50.970 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-09-16T03:47:51.004 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 581 kB/s | 19 kB 00:00 2024-09-16T03:47:51.071 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-09-16T03:47:51.121 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.0 MB/s | 1.0 MB 00:00 2024-09-16T03:47:51.180 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-09-16T03:47:51.205 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-16T03:47:51.230 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 287 kB/s | 31 kB 00:00 2024-09-16T03:47:51.255 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-16T03:47:51.259 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:51.259 INFO:teuthology.orchestra.run.smithi163.stdout:Total 7.7 MB/s | 8.7 MB 00:01 2024-09-16T03:47:51.307 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:47:51.324 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:47:51.324 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:47:51.326 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:51.403 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:47:51.446 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:47:51.492 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:47:51.528 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:47:51.528 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:47:51.531 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:47:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:47:51.750 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:47:51.852 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:47:51.895 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:47:51.923 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:47:51.955 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:47:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:47:52.192 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:47:52.277 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:47:52.370 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:47:52.495 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:47:52.590 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:47:52.641 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:47:52.678 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:47:52.726 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T03:47:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:47:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:47:53.783 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T03:47:53.819 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:47:54.260 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:47:54.261 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:47:54.595 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:47:54.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:54.595 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:47:54.595 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:47:54.596 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:54.597 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:47:54.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:47:55.182 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T03:47:55.182 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:47:55.183 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:47:55.394 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:55.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:47:55.502 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:55.502 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:47:55.502 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:55.503 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:47:55.527 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:47:55.528 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-16T03:47:55.528 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:47:55.528 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-16T03:47:55.528 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:47:55.529 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:47:55.530 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 38 M 2024-09-16T03:47:55.531 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 212 M 2024-09-16T03:47:55.531 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:47:55.710 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-16T03:47:55.760 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T03:47:55.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:47:55.860 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-09-16T03:47:55.910 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-16T03:47:55.961 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-16T03:47:56.161 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2024-09-16T03:47:56.220 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-16T03:47:56.337 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:47:56.387 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-09-16T03:47:56.462 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-16T03:47:56.472 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:47:56.473 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-09-16T03:47:56.474 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-09-16T03:47:56.475 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:47:56.475 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 38 M 2024-09-16T03:47:56.475 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 212 M 2024-09-16T03:47:56.475 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:47:56.487 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-16T03:47:56.771 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): ceph-mgr-diskprediction-local-19.1.1-3 6.2 MB/s | 7.4 MB 00:01 2024-09-16T03:47:56.791 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 880 kB/s | 191 kB 00:00 2024-09-16T03:47:56.849 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 564 kB/s | 33 kB 00:00 2024-09-16T03:47:56.883 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-16T03:47:56.921 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-16T03:47:56.922 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:47:56.922 INFO:teuthology.orchestra.run.smithi070.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-16T03:47:56.933 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 362 kB/s | 18 kB 00:00 2024-09-16T03:47:56.983 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 772 kB/s | 38 kB 00:00 2024-09-16T03:47:57.276 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:47:57.302 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:47:57.302 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:47:57.739 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:47:57.740 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:47:57.817 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-32 6.0 MB/s | 7.4 MB 00:01 2024-09-16T03:47:58.034 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-16T03:47:58.059 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-16T03:47:58.143 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-16T03:47:58.517 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:47:58.591 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:47:58.643 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:47:58.736 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:47:58.771 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:47:58.935 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.5 MB/s | 4.9 MB 00:01 2024-09-16T03:47:58.958 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:47:58.994 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:47:59.072 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:47:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:47:59.824 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:48:00.527 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.0 MB/s | 5.1 MB 00:02 2024-09-16T03:48:01.084 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:48:02.126 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:48:02.153 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:48:04.688 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:48:04.688 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:48:04.689 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:48:05.130 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:05.131 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:05.359 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:07 2024-09-16T03:48:05.360 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:05.361 INFO:teuthology.orchestra.run.smithi163.stdout:Total 4.3 MB/s | 38 MB 00:08 2024-09-16T03:48:05.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:48:05.722 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:05.748 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:05.748 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:06.049 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:16 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:06.183 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:48:06.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:06.185 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:06.186 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2024-09-16T03:48:06.186 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:06.186 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:06.186 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:06.187 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.2 M 2024-09-16T03:48:06.187 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-09-16T03:48:06.187 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:06.463 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 202 kB/s | 19 kB 00:00 2024-09-16T03:48:06.489 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 26 kB 00:00 2024-09-16T03:48:06.522 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 320 kB/s | 49 kB 00:00 2024-09-16T03:48:06.606 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-16T03:48:06.631 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-16T03:48:06.682 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 646 kB/s | 32 kB 00:00 2024-09-16T03:48:06.707 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-09-16T03:48:06.732 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 280 kB/s | 14 kB 00:00 2024-09-16T03:48:06.766 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-16T03:48:06.816 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-16T03:48:06.841 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 803 kB/s | 59 kB 00:00 2024-09-16T03:48:06.866 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T03:48:06.954 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:07.033 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:48:07.050 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-09-16T03:48:07.053 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:07.053 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-16T03:48:07.093 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:48:07.117 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:07.131 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:07.132 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:07.198 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:48:07.220 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:07.220 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:07.242 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:48:07.458 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:48:07.535 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:48:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:48:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:48:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:48:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:48:08.188 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:48:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:48:08.313 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:48:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:48:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:48:09.094 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:48:09.143 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:48:09.176 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:48:09.225 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:48:09.246 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:48:09.575 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:48:10.045 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:48:10.046 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:48:10.382 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:10.383 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:10.625 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:48:10.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:48:10.651 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:48:11.192 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:11.326 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:11.326 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:11.326 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:48:11.326 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-09-16T03:48:11.327 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:11.328 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 7.0 M 2024-09-16T03:48:11.328 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 33 M 2024-09-16T03:48:11.328 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:11.586 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T03:48:11.611 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 633 kB/s | 146 kB 00:00 2024-09-16T03:48:11.661 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T03:48:11.695 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-16T03:48:11.720 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-16T03:48:11.896 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-16T03:48:11.898 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:11.898 INFO:teuthology.orchestra.run.smithi070.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-16T03:48:11.968 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:11.979 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:11.979 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:12.077 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:12.081 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:12.387 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:12.551 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:48:12.605 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:48:13.128 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:48:13.209 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:48:13.264 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:48:13.265 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:48:13.265 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:48:13.265 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:48:13.265 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:48:13.265 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:48:13.272 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:48:13.301 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:48:13.751 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:13.752 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:14.072 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:48:14.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:48:14.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:48:14.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:48:14.073 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:48:14.073 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:48:14.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:48:14.349 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:14.350 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:14.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2024-09-16T03:48:14.739 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:14.873 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:14.874 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:14.874 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:48:14.874 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:14.874 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:14.874 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:48:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.2 M 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-09-16T03:48:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:15.160 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:15.294 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:15.294 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:15.294 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:48:15.294 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:15.294 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-09-16T03:48:15.295 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:15.296 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 976 k 2024-09-16T03:48:15.296 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 2.9 M 2024-09-16T03:48:15.296 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:15.522 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 280 kB/s | 49 kB 00:00 2024-09-16T03:48:15.631 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 90 kB/s | 26 kB 00:00 2024-09-16T03:48:15.656 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.4 MB/s | 896 kB 00:00 2024-09-16T03:48:15.656 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 62 kB/s | 19 kB 00:00 2024-09-16T03:48:15.781 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 208 kB/s | 80 kB 00:00 2024-09-16T03:48:15.781 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:15.782 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.0 MB/s | 976 kB 00:00 2024-09-16T03:48:15.794 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:15.816 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:15.816 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:15.897 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:15.897 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:16.023 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 433 kB/s | 159 kB 00:00 2024-09-16T03:48:16.082 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:16.107 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 380 kB/s | 222 kB 00:00 2024-09-16T03:48:16.132 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 587 kB/s | 279 kB 00:00 2024-09-16T03:48:16.157 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 400 kB/s | 54 kB 00:00 2024-09-16T03:48:16.199 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-16T03:48:16.216 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 168 kB/s | 14 kB 00:00 2024-09-16T03:48:16.216 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:48:16.233 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-16T03:48:16.258 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.5 MB/s | 220 kB 00:00 2024-09-16T03:48:16.280 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:48:16.283 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T03:48:16.325 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-09-16T03:48:16.329 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:16.329 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-09-16T03:48:16.399 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:16.415 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:16.415 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:16.505 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:16.506 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:16.978 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:48:16.978 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:48:17.031 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:17.201 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:48:17.257 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:48:17.257 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:17.257 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:17.257 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:48:17.257 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:17.258 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:17.260 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:48:17.325 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:48:17.361 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:48:17.403 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:48:17.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:48:17.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-volume 2024-09-16T03:48:17.607 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:48:17.663 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:48:18.104 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:18.239 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:18.240 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:18.241 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:18.241 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:18.241 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 285 k 2024-09-16T03:48:18.241 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.3 M 2024-09-16T03:48:18.241 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:18.383 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:48:18.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:48:18.469 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T03:48:18.469 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:18.469 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:48:18.470 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T03:48:18.470 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:18.478 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:18.478 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:18.505 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:18.505 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:18.518 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:48:18.548 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:48:18.722 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:18.805 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:18.821 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:18.821 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:48:18.821 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:48:18.821 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:48:19.336 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:48:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:48:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:48:19.337 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:48:19.675 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:19.676 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:19.850 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:19.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:48:20.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2024-09-16T03:48:20.553 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:20.686 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:20.687 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:20.687 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:48:20.687 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:20.687 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-09-16T03:48:20.688 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:20.689 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 7.0 M 2024-09-16T03:48:20.689 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-09-16T03:48:20.689 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:20.758 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:20.896 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:20.897 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:20.898 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:20.898 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 127 k 2024-09-16T03:48:20.898 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 456 k 2024-09-16T03:48:20.898 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:21.091 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 657 kB/s | 127 kB 00:00 2024-09-16T03:48:21.092 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:21.092 INFO:teuthology.orchestra.run.smithi070.stdout:Total 651 kB/s | 127 kB 00:00 2024-09-16T03:48:21.092 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:21.097 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:21.098 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:21.167 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:21.167 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:21.172 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-16T03:48:21.238 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 613 kB/s | 146 kB 00:00 2024-09-16T03:48:21.335 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:21.347 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-16T03:48:21.398 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2024-09-16T03:48:21.423 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 317 kB/s | 58 kB 00:00 2024-09-16T03:48:21.431 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:21.448 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 758 kB/s | 74 kB 00:00 2024-09-16T03:48:21.451 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:21.452 INFO:teuthology.orchestra.run.smithi163.stdout:Total 9.2 MB/s | 7.0 MB 00:00 2024-09-16T03:48:21.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:21.533 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:21.533 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:21.636 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:21.636 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:21.891 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:21.941 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:22.122 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:48:22.147 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:22.148 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:22.148 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:22.148 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:22.148 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:22.148 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:22.175 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:48:22.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2024-09-16T03:48:22.707 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:48:22.795 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:48:22.858 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:48:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:48:22.970 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:23.062 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:23.107 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:23.107 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:23.107 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:23.203 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2024-09-16T03:48:23.633 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T03:48:23.634 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T03:48:23.634 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:48:23.634 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:48:23.634 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:48:23.634 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:48:23.769 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:23.906 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:23.907 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:23.907 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 32 k 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 142 k 2024-09-16T03:48:23.908 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:23.909 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:24.061 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 207 kB/s | 32 kB 00:00 2024-09-16T03:48:24.062 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:24.062 INFO:teuthology.orchestra.run.smithi070.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-16T03:48:24.063 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:24.066 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:24.066 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:24.086 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:24.086 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:24.177 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:24.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-09-16T03:48:24.284 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:24.675 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:24.777 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:24.906 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:24.913 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:24.914 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 976 k 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.9 M 2024-09-16T03:48:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:25.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2024-09-16T03:48:25.334 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 560 kB/s | 80 kB 00:00 2024-09-16T03:48:25.484 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-16T03:48:25.485 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:25.485 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.7 MB/s | 976 kB 00:00 2024-09-16T03:48:25.496 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:25.518 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:25.519 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:25.599 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:25.599 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:25.659 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:25.750 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:25.792 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:25.793 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:25.794 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:25.794 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:25.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2024-09-16T03:48:25.936 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:48:26.001 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:48:26.429 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:26.519 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:26.563 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:26.564 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:26.564 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:26.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2024-09-16T03:48:26.727 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T03:48:26.728 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:27.004 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:27.196 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:27.287 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:27.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-volume 2024-09-16T03:48:27.330 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:27.331 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:27.332 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:27.395 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2024-09-16T03:48:27.888 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:27.970 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:28.026 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T03:48:28.027 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 285 k 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.3 M 2024-09-16T03:48:28.028 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:28.062 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:28.105 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:28.106 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:28.106 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:28.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2024-09-16T03:48:28.249 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.3 MB/s | 285 kB 00:00 2024-09-16T03:48:28.250 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:28.250 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-16T03:48:28.250 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:28.258 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:28.259 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:28.291 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:28.292 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:28.505 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:28.568 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:28.584 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:48:28.584 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:48:28.584 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:28.730 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:28.821 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:28.864 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:28.865 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:28.865 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:28.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2024-09-16T03:48:29.492 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:29.582 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:29.627 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:29.628 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-16T03:48:29.628 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:29.631 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:29.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2024-09-16T03:48:29.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-09-16T03:48:30.269 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:30.403 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:30.404 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 85 k 2024-09-16T03:48:30.405 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 231 k 2024-09-16T03:48:30.405 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:30.526 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:30.589 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 461 kB/s | 85 kB 00:00 2024-09-16T03:48:30.590 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:30.590 INFO:teuthology.orchestra.run.smithi070.stdout:Total 458 kB/s | 85 kB 00:00 2024-09-16T03:48:30.590 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:30.597 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:30.597 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:30.660 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:30.661 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:30.662 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:30.663 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 127 k 2024-09-16T03:48:30.663 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 456 k 2024-09-16T03:48:30.663 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:30.806 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:30.857 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 653 kB/s | 127 kB 00:00 2024-09-16T03:48:30.857 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:30.857 INFO:teuthology.orchestra.run.smithi163.stdout:Total 648 kB/s | 127 kB 00:00 2024-09-16T03:48:30.857 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:30.863 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:30.863 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:30.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:30.931 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:30.931 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:31.095 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:31.204 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:31.300 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:31.556 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:31.556 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:31.556 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:31.653 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:31.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:31.913 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:32.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-09-16T03:48:32.402 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:32.543 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:32.543 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:32.543 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.1 M 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 13 M 2024-09-16T03:48:32.544 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:32.768 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:32.862 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:32.885 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-16T03:48:32.886 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:32.887 INFO:teuthology.orchestra.run.smithi070.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-16T03:48:32.887 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:32.896 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:32.896 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:32.907 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:32.909 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:32.909 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:32.980 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:32.981 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:32.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-09-16T03:48:33.425 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:33.494 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:33.521 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:33.521 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:48:33.521 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:48:33.521 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:48:33.522 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:48:33.522 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:33.560 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:33.694 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:33.695 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:33.696 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 32 k 2024-09-16T03:48:33.696 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 142 k 2024-09-16T03:48:33.696 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:33.845 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 212 kB/s | 32 kB 00:00 2024-09-16T03:48:33.846 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:33.846 INFO:teuthology.orchestra.run.smithi163.stdout:Total 210 kB/s | 32 kB 00:00 2024-09-16T03:48:33.846 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:33.850 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:33.850 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:33.866 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:33.866 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:33.963 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:34.084 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:34.489 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:34.671 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:34.672 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:34.672 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:34.672 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:34.719 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:34.720 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:34.720 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:34.720 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:34.720 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:34.720 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:34.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2024-09-16T03:48:34.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-09-16T03:48:35.462 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:35.493 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:35.584 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:35.597 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-16T03:48:35.597 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-16T03:48:35.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:35.599 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 170 k 2024-09-16T03:48:35.599 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 494 k 2024-09-16T03:48:35.599 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-16T03:48:35.629 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:35.630 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:35.630 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:35.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-09-16T03:48:35.806 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 824 kB/s | 170 kB 00:00 2024-09-16T03:48:35.806 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:35.807 INFO:teuthology.orchestra.run.smithi070.stdout:Total 818 kB/s | 170 kB 00:00 2024-09-16T03:48:35.807 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-16T03:48:35.815 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-16T03:48:35.815 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-16T03:48:35.878 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-16T03:48:35.878 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-16T03:48:36.031 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-16T03:48:36.120 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:36.278 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:36.374 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:36.413 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:36.414 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:36.415 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:36.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-09-16T03:48:36.535 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:36.802 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:36.802 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:36.802 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-16T03:48:36.802 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:36.803 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:36.803 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-16T03:48:37.001 DEBUG:teuthology.parallel:result is None 2024-09-16T03:48:37.060 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:37.150 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:37.193 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:37.194 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:37.194 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:37.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-09-16T03:48:37.851 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:37.943 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:37.987 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:37.988 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:37.988 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:38.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-09-16T03:48:38.638 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:38.729 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:38.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-09-16T03:48:39.426 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:39.517 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T03:48:39.562 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:39.562 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-09-16T03:48:39.563 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:39.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-09-16T03:48:40.222 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:40.359 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:40.360 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:40.361 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:40.361 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 85 k 2024-09-16T03:48:40.361 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 231 k 2024-09-16T03:48:40.361 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:40.549 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 453 kB/s | 85 kB 00:00 2024-09-16T03:48:40.549 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi163.stdout:Total 450 kB/s | 85 kB 00:00 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:40.556 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:40.556 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:40.620 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:40.620 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:40.765 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:41.510 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:41.792 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-09-16T03:48:42.403 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:42.543 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:42.544 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.1 M 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 13 M 2024-09-16T03:48:42.545 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:42.888 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.0 MB/s | 3.1 MB 00:00 2024-09-16T03:48:42.888 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:42.889 INFO:teuthology.orchestra.run.smithi163.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-16T03:48:42.889 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:42.898 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:42.898 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:42.980 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:42.980 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:43.418 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:43.472 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:43.498 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:43.498 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:48:43.498 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:48:43.498 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:48:43.499 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:48:43.499 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:44.896 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-09-16T03:48:45.474 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:55 ago on Mon 16 Sep 2024 03:46:50 AM UTC. 2024-09-16T03:48:45.610 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-09-16T03:48:45.611 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:45.612 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 170 k 2024-09-16T03:48:45.612 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 494 k 2024-09-16T03:48:45.612 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-09-16T03:48:45.810 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 861 kB/s | 170 kB 00:00 2024-09-16T03:48:45.811 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:48:45.812 INFO:teuthology.orchestra.run.smithi163.stdout:Total 854 kB/s | 170 kB 00:00 2024-09-16T03:48:45.812 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-09-16T03:48:45.819 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-09-16T03:48:45.819 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-09-16T03:48:45.882 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-09-16T03:48:45.883 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-09-16T03:48:46.032 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-09-16T03:48:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:46.608 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T03:48:46.868 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:46.869 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-09-16T03:48:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T03:48:46.869 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:46.869 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-09-16T03:48:47.107 DEBUG:teuthology.parallel:result is None 2024-09-16T03:48:47.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:48:47.255 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:48:47.280 INFO:teuthology.orchestra.run.smithi070.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:48:47.281 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:48:47.281 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:48:47.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:48:47.418 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:48:47.440 INFO:teuthology.orchestra.run.smithi163.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T03:48:47.440 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T03:48:47.440 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T03:48:47.442 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:48:47.442 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:47.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:48:47.473 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:47.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:48:47.506 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:48:47.506 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:47.506 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:48:47.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:48:47.608 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:47.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:48:47.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:48:47.697 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:48:47.697 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:47.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:48:47.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:48:47.789 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:47.789 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:48:47.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:48:47.888 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:48:47.889 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:47.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:48:47.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:48:47.979 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:47.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:48:48.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:48:48.070 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:48:48.158 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:48:48.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-09-16T03:48:48.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2024-09-16T03:48:48.185 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:48:48.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:48:48.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:48:48.253 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:48:48.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:48:48.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:48:48.338 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:48:48.338 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T03:48:48.338 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920', 'branch': 'squid', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905920', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbV6wB7THjGtLpf0taXTtshOVbOHYhAjEIfLUS7sLcqzJMhonZcrEFTyUJCY3ETyirKkKVgIWj9c9IgQWhV9AY=', 'smithi163.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPM4s4lawqQsXThQpAFIXzWhagoeL3dzAlhJPqCxANQiLVUH0Q/d8oW8kG9TULPdB2EABd4VLC+21YEx9cBMlTc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:48:48.339 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:48:48.356 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:48:48.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:48:48.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:48:48.439 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:48.439 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:48:48.490 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:48:48.490 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-09-16T03:48:48.544 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-16 03:48:33.789401601 +0000 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-16 03:46:12.197983995 +0000 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-16 03:46:12.197983995 +0000 2024-09-16T03:48:48.545 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-16 03:46:12.197983995 +0000 2024-09-16T03:48:48.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:48:48.607 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-16T03:48:48.607 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-16T03:48:48.607 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 7.6737e-05 s, 6.7 MB/s 2024-09-16T03:48:48.608 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:48:48.664 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-09-16T03:48:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:48:48.719 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:48.719 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-16 03:48:33.790401571 +0000 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-16 03:46:12.510974694 +0000 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-16 03:46:12.510974694 +0000 2024-09-16T03:48:48.720 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-16 03:46:12.510974694 +0000 2024-09-16T03:48:48.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:48:48.782 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-16T03:48:48.782 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-16T03:48:48.782 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000100163 s, 5.1 MB/s 2024-09-16T03:48:48.783 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:48:48.839 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-09-16T03:48:48.894 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-16 03:48:33.790401571 +0000 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-16 03:46:12.823965392 +0000 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-16 03:46:12.823965392 +0000 2024-09-16T03:48:48.895 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-16 03:46:12.823965392 +0000 2024-09-16T03:48:48.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:48:48.957 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-16T03:48:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-16T03:48:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.2229e-05 s, 6.2 MB/s 2024-09-16T03:48:48.959 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:48:49.016 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-09-16T03:48:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:48:49.072 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-16 03:48:33.790401571 +0000 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-16 03:46:13.167955170 +0000 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-16 03:46:13.167955170 +0000 2024-09-16T03:48:49.073 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-16 03:46:13.167955170 +0000 2024-09-16T03:48:49.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:48:49.135 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-16T03:48:49.136 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-16T03:48:49.136 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000100818 s, 5.1 MB/s 2024-09-16T03:48:49.137 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:48:49.193 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:49.194 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:49.194 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:48:49.210 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:48:49.210 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-09-16T03:48:49.266 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 03:48:43.773587079 +0000 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 03:46:11.864780889 +0000 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 03:46:11.864780889 +0000 2024-09-16T03:48:49.267 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 03:46:11.864780889 +0000 2024-09-16T03:48:49.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:48:49.330 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T03:48:49.330 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T03:48:49.330 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000208759 s, 2.5 MB/s 2024-09-16T03:48:49.331 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:48:49.388 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-09-16T03:48:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:48:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:49.444 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T03:48:49.444 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:49.445 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:49.445 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 03:48:43.774587057 +0000 2024-09-16T03:48:49.445 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 03:46:12.196773380 +0000 2024-09-16T03:48:49.445 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 03:46:12.196773380 +0000 2024-09-16T03:48:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 03:46:12.196773380 +0000 2024-09-16T03:48:49.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:48:49.507 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T03:48:49.508 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T03:48:49.508 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000198856 s, 2.6 MB/s 2024-09-16T03:48:49.509 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:48:49.566 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 03:48:43.774587057 +0000 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 03:46:12.517766119 +0000 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 03:46:12.517766119 +0000 2024-09-16T03:48:49.621 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 03:46:12.517766119 +0000 2024-09-16T03:48:49.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:48:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T03:48:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T03:48:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000263659 s, 1.9 MB/s 2024-09-16T03:48:49.686 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:48:49.743 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-09-16 03:48:43.774587057 +0000 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-09-16 03:46:12.844758723 +0000 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-09-16 03:46:12.844758723 +0000 2024-09-16T03:48:49.799 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-09-16 03:46:12.844758723 +0000 2024-09-16T03:48:49.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:48:49.863 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-09-16T03:48:49.863 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-09-16T03:48:49.863 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000229884 s, 2.2 MB/s 2024-09-16T03:48:49.865 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:48:49.920 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:49.921 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:48:49.921 INFO:tasks.ceph:Generating config... 2024-09-16T03:48:49.922 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T03:48:49.923 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T03:48:49.923 INFO:tasks.ceph:[client] debug objecter = 20 2024-09-16T03:48:49.923 INFO:tasks.ceph:[client] debug rados = 20 2024-09-16T03:48:49.923 INFO:tasks.ceph:[global] lockdep = True 2024-09-16T03:48:49.923 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T03:48:49.923 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T03:48:49.923 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T03:48:49.923 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T03:48:49.924 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T03:48:49.924 INFO:tasks.ceph:[global] ms type = async 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:48:49.924 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T03:48:49.925 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-16T03:48:49.925 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T03:48:49.925 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T03:48:49.925 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T03:48:49.926 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:48:49.927 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:48:49.928 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T03:48:49.928 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:48:49.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:48:49.996 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:48:49.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:48:50.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:48:50.106 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70'), ('mon.c', '[v2:172.21.15.70:3301,v1:172.21.15.70:6790]'), ('mon.b', '172.21.15.163')] 2024-09-16T03:48:50.106 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70,[v2:172.21.15.70:3301,v1:172.21.15.70:6790],172.21.15.163', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T03:48:50.107 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:50.107 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:48:50.162 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --addv c '[v2:172.21.15.70:3301,v1:172.21.15.70:6790]' --add b 172.21.15.163 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:50.247 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:50.247 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:48:50.247 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-09-16T03:48:50.247 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 17 (quincy) 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 3 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:48:50.248 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:48:50.249 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:48:50.265 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 33825cf0-3459-43e2-ae47-46b331d213a2... 2024-09-16T03:48:50.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:48:50.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:48:50.348 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-09-16T03:48:50.348 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-09-16T03:48:50.348 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:48:50.348 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-09-16T03:48:50.348 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-09-16T03:48:50.349 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:48:50.350 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.351 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-09-16T03:48:50.352 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-09-16T03:48:50.353 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = "172.21.15.70,[v2:172.21.15.70:3301,v1:172.21.15.70:6790],172.21.15.163" 2024-09-16T03:48:50.353 INFO:teuthology.orchestra.run.smithi070.stdout: lockdep = True 2024-09-16T03:48:50.353 INFO:teuthology.orchestra.run.smithi070.stdout: mon client directed command retry = 5 2024-09-16T03:48:50.353 INFO:teuthology.orchestra.run.smithi070.stdout: mon election default strategy = 3 2024-09-16T03:48:50.353 INFO:teuthology.orchestra.run.smithi070.stdout: ms bind msgr1 = False 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: ms bind msgr2 = True 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: ms inject socket failures = 5000 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: ms type = async 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.354 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-09-16T03:48:50.355 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: bdev async discard = True 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: bdev enable discard = True 2024-09-16T03:48:50.356 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore allocator = bitmap 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore zero block detection = True 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:48:50.357 INFO:teuthology.orchestra.run.smithi070.stdout: osd class default list = * 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd class load list = * 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd client watch timeout = 120 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify cached snaps = True 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify missing on start = True 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock override recovery settings = True 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue = debug_random 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue cut off = debug_random 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2024-09-16T03:48:50.358 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-16T03:48:50.359 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:48:50.360 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon min osdmap epochs = 50 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune interval = 2 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune min = 15 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon scrub interval = 300 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on pool no app = False 2024-09-16T03:48:50.361 INFO:teuthology.orchestra.run.smithi070.stdout: paxos service trim min = 10 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: debug objecter = 20 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout: debug rados = 20 2024-09-16T03:48:50.362 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-09-16T03:48:50.363 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.c] 2024-09-16T03:48:50.363 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.b] 2024-09-16T03:48:50.364 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-09-16T03:48:50.364 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2024-09-16T03:48:50.364 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:48:50.364 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2024-09-16T03:48:50.365 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.366 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:48:50.367 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon_allow_pool_size_one = true 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2024-09-16T03:48:50.368 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.70,[v2:172.21.15.70:3301,v1:172.21.15.70:6790],172.21.15.163" 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: lockdep = True 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: mon client directed command retry = 5 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: mon election default strategy = 3 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: ms bind msgr1 = False 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: ms bind msgr2 = True 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: ms inject socket failures = 5000 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: ms type = async 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2024-09-16T03:48:50.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug trim objects = true 2024-09-16T03:48:50.370 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2024-09-16T03:48:50.371 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bdev async discard = True 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bdev enable discard = True 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore allocator = bitmap 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore block size = 96636764160 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore fsck on mount = True 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore zero block detection = True 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluefs = 20 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluestore = 20 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: debug monc = 20 2024-09-16T03:48:50.372 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: debug rocksdb = 10 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd full ratio = 0.9 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd class default list = * 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd class load list = * 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd client watch timeout = 120 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify cached snaps = True 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify missing on start = True 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock override recovery settings = True 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = bluestore 2024-09-16T03:48:50.373 INFO:teuthology.orchestra.run.smithi163.stdout: osd op queue = debug_random 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: osd op queue cut off = debug_random 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: osd shutdown pgref assert = True 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2024-09-16T03:48:50.374 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:48:50.375 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: # 1m isn't quite enough 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon min osdmap epochs = 50 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune interval = 2 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune min = 15 2024-09-16T03:48:50.376 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: mon scrub interval = 300 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on pool no app = False 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: paxos service trim min = 10 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: debug monc = 20 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: debug objecter = 20 2024-09-16T03:48:50.377 INFO:teuthology.orchestra.run.smithi163.stdout: debug rados = 20 2024-09-16T03:48:50.378 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2024-09-16T03:48:50.378 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2024-09-16T03:48:50.378 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2024-09-16T03:48:50.378 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:48:50.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:48:50.459 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:48:50.459 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:50.460 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:48:50.475 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:50.475 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:48:50.531 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:48:50.531 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:50.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:48:50.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:48:50.605 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:48:50.606 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:50.661 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:48:50.661 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:50.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:48:50.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:48:50.697 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:48:50.697 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:50.752 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:48:50.752 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:48:50.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:48:50.803 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:48:50.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:48:50.865 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:48:50.868 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:48:50.869 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:48:50.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:48:50.910 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:48:50.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T03:48:50.964 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T03:48:50.977 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:48:50.977 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:48:50.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:48:51.001 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:51.001 INFO:tasks.ceph:role: osd.0 2024-09-16T03:48:51.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:48:51.001 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:48:51.065 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:48:51.065 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:48:51.066 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:48:51.098 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-16T03:48:51.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-16T03:48:51.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:48:51.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:48:51.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:48:51.303 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:51.304 INFO:tasks.ceph:role: osd.1 2024-09-16T03:48:51.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:48:51.304 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:48:51.367 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:48:51.367 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:48:51.367 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:48:51.368 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:48:51.400 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-16T03:48:51.401 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-16T03:48:51.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T03:48:51.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:48:51.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T03:48:51.604 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:51.604 INFO:tasks.ceph:role: osd.2 2024-09-16T03:48:51.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:48:51.605 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:48:51.669 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:48:51.670 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:48:51.701 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-16T03:48:51.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-16T03:48:51.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T03:48:51.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T03:48:51.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T03:48:51.907 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T03:48:51.907 INFO:tasks.ceph:role: osd.3 2024-09-16T03:48:51.907 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T03:48:51.907 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:48:51.971 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:48:51.972 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:48:51.972 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:48:51.972 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:48:51.972 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:48:52.005 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-16T03:48:52.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-16T03:48:52.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T03:48:52.076 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T03:48:52.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:52.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:52.236+0000 7f658db89640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:48:52.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:52.236+0000 7f658db89640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:48:52.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:52.236+0000 7f658db89640 -1 bdev(0x560ca10af800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:48:52.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:52.236+0000 7f658db89640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:48:56.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:48:56.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:56.275+0000 7f1e2b822640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:56.275+0000 7f1e2b822640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:56.275+0000 7f1e2b822640 -1 bdev(0x56458524d800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:48:56.275+0000 7f1e2b822640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:48:59.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:48:59.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:00.066+0000 7f0702685640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T03:49:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:00.067+0000 7f0702685640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T03:49:00.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:00.067+0000 7f0702685640 -1 bdev(0x55806852f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:00.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:00.067+0000 7f0702685640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T03:49:03.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T03:49:03.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:03.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:03.826+0000 7f6218414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T03:49:03.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:03.826+0000 7f6218414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T03:49:03.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:03.827+0000 7f6218414640 -1 bdev(0x5639e8d5b800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:03.827+0000 7f6218414640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T03:49:07.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T03:49:07.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:49:07.539 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T03:49:07.539 INFO:tasks.ceph:role: osd.4 2024-09-16T03:49:07.539 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:49:07.539 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:49:07.616 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:49:07.616 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:49:07.617 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:49:08.129 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T03:49:08.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T03:49:08.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T03:49:08.233 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:49:08.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:49:08.335 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T03:49:08.335 INFO:tasks.ceph:role: osd.5 2024-09-16T03:49:08.335 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:49:08.335 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:49:08.410 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:49:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:49:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:49:08.411 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:49:08.879 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T03:49:08.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T03:49:08.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T03:49:08.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:49:09.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T03:49:09.069 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T03:49:09.069 INFO:tasks.ceph:role: osd.6 2024-09-16T03:49:09.069 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:49:09.070 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:49:09.141 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:49:09.142 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:49:09.620 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T03:49:09.622 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T03:49:09.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T03:49:09.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T03:49:09.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T03:49:09.867 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T03:49:09.867 INFO:tasks.ceph:role: osd.7 2024-09-16T03:49:09.867 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T03:49:09.867 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:49:09.940 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:49:10.455 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-09-16T03:49:10.457 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-09-16T03:49:10.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T03:49:10.604 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T03:49:10.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:10.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:10.738+0000 7fb2b340b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:49:10.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:10.739+0000 7fb2b340b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:49:10.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:10.739+0000 7fb2b340b640 -1 bdev(0x5604afaf3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:10.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:10.739+0000 7fb2b340b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:49:14.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:49:14.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:14.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:14.520+0000 7facedf3e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:49:14.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:14.521+0000 7facedf3e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:49:14.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:14.521+0000 7facedf3e640 -1 bdev(0x55a378303800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:14.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:14.521+0000 7facedf3e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:49:18.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:49:18.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:18.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:18.567+0000 7fadd1a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T03:49:18.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:18.567+0000 7fadd1a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T03:49:18.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:18.567+0000 7fadd1a0f640 -1 bdev(0x557cfa62b800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:18.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:18.567+0000 7fadd1a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T03:49:22.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T03:49:22.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:22.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:22.592+0000 7fa8c660b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T03:49:22.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:22.592+0000 7fa8c660b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T03:49:22.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:22.592+0000 7fa8c660b640 -1 bdev(0x55e893879800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T03:49:22.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:22.592+0000 7fa8c660b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T03:49:26.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T03:49:26.281 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T03:49:26.281 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T03:49:26.314 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T03:49:26.383 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T03:49:26.451 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T03:49:26.518 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.519 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T03:49:26.585 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T03:49:26.614 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T03:49:26.683 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T03:49:26.751 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.751 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T03:49:26.819 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T03:49:26.885 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:26.885 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T03:49:26.904 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:26.904 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T03:49:26.941 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T03:49:26.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:49:26.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.y] 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCiqudmWJrYLxAAiPvJprIHIp1D9TwWrXWE3g== 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCkqudmEo4sDhAA8Vzy/qQsBPUfxnv17Uujlg== 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCoqudmVmyFEBAAwu5N1i2sTsJ4xTbwZDkIEw== 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCsqudmP4ATBBAAdjq12498dbe6N4IkQI/mnA== 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-09-16T03:49:26.985 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCvqudmZl5fMRAAZnRoIy3M0XHDInk9R8XfWA== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCiqudmg+iNMxAAlcIkctRCbfEoCO5maHlVUg== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.4] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQC2qudmllckLBAAwJFC6+DoTAV0fekQsovZZg== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.5] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQC6qudmG70lHxAAcut3q4RG3tPgOAf82RDAeg== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.6] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQC+qudmdKfrIRAAgxfADAt+Sw27m4RmhRs3Eg== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.7] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDCqudmN39nIxAAbj8nem53hEUypOU+SzXd+Q== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCiqudmlwAwNhAA8DsknEkfz6kaDFa4byJ6LA== 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout:[client.1] 2024-09-16T03:49:26.986 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQCiqudmwP5nORAAI7G8YP/bkTsld5XVdmP9/Q== 2024-09-16T03:49:27.005 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.y] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCiqudmWJrYLxAAiPvJprIHIp1D9TwWrXWE3g== 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.0] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCkqudmEo4sDhAA8Vzy/qQsBPUfxnv17Uujlg== 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.1] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCoqudmVmyFEBAAwu5N1i2sTsJ4xTbwZDkIEw== 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.2] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCsqudmP4ATBBAAdjq12498dbe6N4IkQI/mnA== 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.3] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCvqudmZl5fMRAAZnRoIy3M0XHDInk9R8XfWA== 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.x] 2024-09-16T03:49:27.006 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCiqudmg+iNMxAAlcIkctRCbfEoCO5maHlVUg== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.4] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQC2qudmllckLBAAwJFC6+DoTAV0fekQsovZZg== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.5] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQC6qudmG70lHxAAcut3q4RG3tPgOAf82RDAeg== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.6] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQC+qudmdKfrIRAAgxfADAt+Sw27m4RmhRs3Eg== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.7] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQDCqudmN39nIxAAbj8nem53hEUypOU+SzXd+Q== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCiqudmlwAwNhAA8DsknEkfz6kaDFa4byJ6LA== 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout:[client.1] 2024-09-16T03:49:27.007 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCiqudmwP5nORAAI7G8YP/bkTsld5XVdmP9/Q== 2024-09-16T03:49:27.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:49:27.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:49:27.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:49:27.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:49:27.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:49:27.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:49:27.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:49:27.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:49:27.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:49:27.692 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T03:49:27.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T03:49:27.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:49:27.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T03:49:27.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T03:49:27.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:49:27.996 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T03:49:28.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T03:49:28.046 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:49:28.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T03:49:28.168 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:28.171 DEBUG:teuthology.orchestra.run.smithi163:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:49:28.223 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T03:49:28.224 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:49:28.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:49:28.226 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:49:28.226 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:49:28.227 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:49:28.229 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:49:28.229 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:49:28.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:49:28.266 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:49:28.266 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T03:49:28.266 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T03:49:28.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T03:49:28.269 INFO:tasks.ceph.mgr.y:Started 2024-09-16T03:49:28.269 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T03:49:28.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T03:49:28.271 INFO:tasks.ceph.mgr.x:Started 2024-09-16T03:49:28.271 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T03:49:28.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.369+0000 7f6a181bd640 10 monclient: get_monmap_and_config 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.369+0000 7f6a181bd640 10 monclient: build_initial_monmap 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.369+0000 7f6a181bd640 10 monclient: monmap: 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:28.372 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:28.370718+0000 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:28.370718+0000 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.369+0000 7f6a181bd640 1 Processor -- start 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 -- start start 2024-09-16T03:49:28.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient: init 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a100b5650 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-c con 0x7f6a100b3120 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:28.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-b con 0x7f6a10146a40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a100aae40 con 0x7f6a100b5650 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f6a100b5a30 con 0x7f6a100b3120 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f6a10140b20 con 0x7f6a10146a40 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a181bd640 10 monclient(hunting): authenticate will time out at 727.032688s 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a15f59640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6a100b5650 0x7f6a10140410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43580/0 (socket says 172.21.15.70:43580) 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a15f59640 1 -- 172.21.15.70:0/1788071277 learned_addr learned my addr 172.21.15.70:0/1788071277 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f6a10146a40 auth_method 0 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.370+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f6a100b3120 auth_method 0 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f6a10146a40 auth_method 0 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.571+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:28.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f6a100b3120 auth_method 0 2024-09-16T03:49:28.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f6a10146a40 auth_method 0 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.972+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f6a100b3120 auth_method 0 2024-09-16T03:49:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:28.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f6a10146a40 auth_method 0 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:29.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.773+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f6a100b3120 auth_method 0 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:29.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:29.774+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): tick 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 msgr2=0x7f6a1014d890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:31.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a10146a40 0x7f6a1014d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 msgr2=0x7f6a10146500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f6a100b3120 0x7f6a10146500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 >> v1:172.21.15.70:6789/0 conn(0x7f6a100b5650 legacy=0x7f6a10140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69f40036e0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): picked mon.noname-a con 0x7f69f4007370 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a100b5cd0 con 0x7f69f40036e0 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6a100b5a30 con 0x7f69f4007370 2024-09-16T03:49:31.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a077fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69f4007370 auth_method 0 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:31.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69f4007370 auth_method 0 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:31.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.572+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69f4007370 auth_method 0 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:31.973+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:32.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:32.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69f4007370 auth_method 0 2024-09-16T03:49:32.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:32.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:32.775+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:49:34.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): tick 2024-09-16T03:49:34.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:49:34.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4007370 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 >> v1:172.21.15.163:6789/0 conn(0x7f69f40036e0 legacy=0x7f69f4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 reap_dead start 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69f4009fc0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 0x7f69f4009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): picked mon.noname-c con 0x7f69f4006270 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 0x7f69f4005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): picked mon.noname-a con 0x7f69f4006650 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69f400af40 con 0x7f69f4009fc0 2024-09-16T03:49:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6a100b5a30 con 0x7f69f4006650 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 1 -- 172.21.15.70:0/1788071277 --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f6a10140b20 con 0x7f69f4006270 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a077fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 0x7f69f4005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 0x7f69f4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f69f4006650 auth_method 0 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69f4006270 auth_method 0 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 msgr2=0x7f69f4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 msgr2=0x7f69f4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 0x7f69f4005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 0x7f69f4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 0x7f69f4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.371+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:34.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3107941577 0 0) 0x7f69f400af40 con 0x7f69f4009fc0 2024-09-16T03:49:34.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 10 monclient(hunting): _negotiate 2024-09-16T03:49:34.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 10 monclient(hunting): my global_id is 4102 2024-09-16T03:49:34.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.541+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69fc003520 con 0x7f69f4009fc0 2024-09-16T03:49:34.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1577334916 0 0) 0x7f69fc003520 con 0x7f69f4009fc0 2024-09-16T03:49:34.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 msgr2=0x7f69f4009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:34.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.163:3300/0 conn(0x7f69f4006270 0x7f69f4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 msgr2=0x7f69f4005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 --2- 172.21.15.70:0/1788071277 >> v2:172.21.15.70:3300/0 conn(0x7f69f4006650 0x7f69f4005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _finish_hunting 0 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.163:6789/0 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f4006b00 con 0x7f69f4009fc0 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _finish_auth 0 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _check_auth_tickets 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f6a0c0016b0 con 0x7f69f4009fc0 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a181bd640 5 monclient: authenticate success, global_id 4102 2024-09-16T03:49:34.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a181bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: dump: 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] but i'm connected to v1:172.21.15.163:6789/0, reconnecting 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 >> v1:172.21.15.163:6789/0 conn(0x7f69f4009fc0 legacy=0x7f69f400a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69fc0039b0 0x7f69fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): picked mon.a con 0x7f69fc0039b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc004800 0x7f69fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): picked mon.b con 0x7f69fc004800 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 0x7f69fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): picked mon.c con 0x7f69fc005210 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100b5a30 con 0x7f69fc0039b0 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a10140b20 con 0x7f69fc004800 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f69fc00c1c0 con 0x7f69fc005210 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a14f57640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a181bd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15f59640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69fc0039b0 0x7f69fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 0x7f69fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15f59640 10 monclient(hunting): get_auth_request con 0x7f69fc0039b0 auth_method 0 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15f59640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15f59640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f69fc005210 auth_method 0 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a1675a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15758640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc004800 0x7f69fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f69fc004800 auth_method 0 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.545+0000 7f6a15758640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc004800 msgr2=0x7f69fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc004800 0x7f69fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69fc0039b0 msgr2=0x7f69fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69fc0039b0 0x7f69fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient: _finish_hunting 0 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 monclient: found mon.c 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:34.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc003520 con 0x7f69fc005210 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient: _finish_auth 0 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a1675a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.546+0000 7f6a15f59640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69fc0039b0 0x7f69fc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.547+0000 7f6a1675a640 1 -- 172.21.15.70:0/1788071277 reap_dead start 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.547+0000 7f6a1675a640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 0x7f69fc00b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a10146e20 tx=0x7f69fc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa5e0ae2fec52191 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c007c00 con 0x7f69fc005210 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: dump: 2024-09-16T03:49:34.557 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a0802f070 con 0x7f69fc005210 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 1 -- 172.21.15.70:0/1788071277 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08031070 con 0x7f69fc005210 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:34.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a14f57640 10 monclient: dump: 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a181bd640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:34.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.548+0000 7f6a181bd640 4 set_mon_vals no callback set 2024-09-16T03:49:34.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 10 monclient: shutdown 2024-09-16T03:49:34.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:34.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 -- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 msgr2=0x7f69fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 0x7f69fc00b9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a10146e20 tx=0x7f69fc005830 comp rx=0 tx=0).stop 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 -- 172.21.15.70:0/1788071277 shutdown_connections 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1788071277 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f69fc005210 0x7f69fc00b9c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 -- 172.21.15.70:0/1788071277 >> 172.21.15.70:0/1788071277 conn(0x7f6a100679c0 msgr2=0x7f6a10067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 -- 172.21.15.70:0/1788071277 shutdown_connections 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.549+0000 7f6a181bd640 1 -- 172.21.15.70:0/1788071277 wait complete. 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.550+0000 7f6a181bd640 10 monclient: build_initial_monmap 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.550+0000 7f6a181bd640 10 monclient: monmap: 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 librados: starting msgr at 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 librados: starting objecter 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 Processor -- start 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 -- start start 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 librados: setting wanted keys 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 1 librados: calling monclient init 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.551+0000 7f6a181bd640 10 monclient: init 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:34.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-b con 0x7f69fc005210 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100afd20 0x7f6a1013ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-a con 0x7f6a100afd20 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a1013d3d0 0x7f6a101d8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): picked mon.noname-c con 0x7f6a1013d3d0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a10146e90 con 0x7f6a100afd20 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a100aac00 con 0x7f69fc005210 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f6a10147010 con 0x7f6a1013d3d0 2024-09-16T03:49:34.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a1013d3d0 0x7f6a101d8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:42544/0 (socket says 172.21.15.70:42544) 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a181bd640 10 monclient(hunting): authenticate will time out at 733.214685s 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 1 -- 172.21.15.70:0/1834077441 learned_addr learned my addr 172.21.15.70:0/1834077441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): get_auth_request con 0x7f69fc005210 auth_method 0 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): get_auth_request con 0x7f6a1013d3d0 auth_method 0 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100afd20 0x7f6a1013ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): get_auth_request con 0x7f6a100afd20 auth_method 0 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15f59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a15758640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:34.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.552+0000 7f6a1675a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 -- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a1013d3d0 msgr2=0x7f6a101d8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a1013d3d0 0x7f6a101d8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 -- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100afd20 msgr2=0x7f6a1013ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100afd20 0x7f6a1013ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 10 monclient: _finish_hunting 0 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 monclient: found mon.noname-b 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 -- 172.21.15.70:0/1834077441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a100ad260 con 0x7f69fc005210 2024-09-16T03:49:34.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a15f59640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0072c0 tx=0x7f6a0c007780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a067fc640 10 client.?.objecter ms_handle_connect 0x7f69fc005210 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.553+0000 7f6a067fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c001650 con 0x7f69fc005210 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: dump: 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:34.571 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a0c018070 con 0x7f69fc005210 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00fb60 con 0x7f69fc005210 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:34.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 10 monclient: dump: 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:34.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 5 monclient: authenticate success, global_id 4108 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 10 monclient: _renew_subs 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a100b1d30 con 0x7f69fc005210 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a07fff640 4 set_mon_vals no callback set 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 10 monclient: _renew_subs 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a100b0140 con 0x7f69fc005210 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a181bd640 1 librados: init done 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.554+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a0c00fd00 con 0x7f69fc005210 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.555+0000 7f6a181bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.555+0000 7f6a181bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.555+0000 7f6a181bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:34.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.556+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1013d7b0 con 0x7f69fc005210 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.557+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a0c01d5a0 con 0x7f69fc005210 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.557+0000 7f6a067fc640 10 client.4108.objecter ms_dispatch 0x7f6a10000b90 osd_map(1..1 src has 1..1) 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.558+0000 7f6a067fc640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.558+0000 7f6a067fc640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.558+0000 7f6a067fc640 20 client.4108.objecter dump_active .. 0 homeless 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.560+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f6a0c01d7a0 con 0x7f69fc005210 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.560+0000 7f6a067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:34.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.560+0000 7f6a067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:34.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.624+0000 7f6a181bd640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:49:34.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.624+0000 7f6a181bd640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:49:34.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.624+0000 7f6a181bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:34.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.624+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6a100b9e90 con 0x7f69fc005210 2024-09-16T03:49:34.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.631+0000 7f6a067fc640 1 -- 172.21.15.70:0/1834077441 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6a0c01d980 con 0x7f69fc005210 2024-09-16T03:49:34.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.631+0000 7f6a067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:49:34.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.631+0000 7f6a067fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T03:49:34.634 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 10 librados: watch_flush enter 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 10 librados: watch_flush exit 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 20 client.4108.objecter shutdown clearing up homeless session... 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 10 client.4108.objecter successfully canceled tick 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 10 monclient: shutdown 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:34.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 msgr2=0x7f6a100b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0072c0 tx=0x7f6a0c007780 comp rx=0 tx=0).stop 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 shutdown_connections 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6a1013d3d0 0x7f6a101d8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a100afd20 0x7f6a1013ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 --2- 172.21.15.70:0/1834077441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69fc005210 0x7f6a100b0770 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 >> 172.21.15.70:0/1834077441 conn(0x7f6a100679c0 msgr2=0x7f6a100b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 shutdown_connections 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 -- 172.21.15.70:0/1834077441 wait complete. 2024-09-16T03:49:34.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:34.632+0000 7f6a181bd640 1 librados: shutdown 2024-09-16T03:49:34.646 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T03:49:34.646 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T03:49:34.646 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:34.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T03:49:34.663 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:34.661+0000 7f37f6412040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T03:49:34.671 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:34.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T03:49:34.707 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:34.704+0000 7f37f6412040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T03:49:34.718 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:34.717+0000 7f14277b9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T03:49:34.731 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:34.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T03:49:34.761 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:34.759+0000 7f14277b9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T03:49:34.797 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:49:34.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T03:49:34.859 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:34.857+0000 7f37f6412040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T03:49:34.861 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:34.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T03:49:34.886 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:34.886 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T03:49:34.911 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:34.910+0000 7f14277b9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T03:49:34.912 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:34.909+0000 7f37f6412040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T03:49:34.935 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:34.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T03:49:34.965 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:34.963+0000 7f14277b9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T03:49:34.989 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-09-16T03:49:34.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T03:49:35.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 0e4116b6-869c-4008-a494-02c6bf40f9a1 0 2024-09-16T03:49:35.072 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:35.070+0000 7f37f6412040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:49:35.143 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:35.141+0000 7f14277b9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:49:35.183 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:35.180+0000 7f37f6412040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T03:49:35.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 10 monclient: get_monmap_and_config 2024-09-16T03:49:35.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 10 monclient: monmap: 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:35.193757+0000 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:35.193757+0000 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:35.195 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 1 Processor -- start 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 1 -- start start 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.193+0000 7f5fb7808640 10 monclient: init 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5fb00b28f0 0x7f5fb0070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb00b28f0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fb0141940 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:35.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5fb013e630 0x7f5fb0071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb013e630 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb01411a0 con 0x7f5fb0141940 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00b2cd0 con 0x7f5fb013e630 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00735a0 con 0x7f5fb00b28f0 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb7808640 10 monclient(hunting): authenticate will time out at 750.762628s 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5fb00b28f0 0x7f5fb0070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5fb00b28f0 0x7f5fb0070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39730/0 (socket says 172.21.15.163:39730) 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 1 -- 172.21.15.163:0/2783876786 learned_addr learned my addr 172.21.15.163:0/2783876786 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): get_auth_request con 0x7f5fb00b28f0 auth_method 0 2024-09-16T03:49:35.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb55a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb5da5640 1 --2- 172.21.15.163:0/2783876786 >> v2:172.21.15.70:3300/0 conn(0x7f5fb013e630 0x7f5fb0071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb5da5640 10 monclient(hunting): get_auth_request con 0x7f5fb013e630 auth_method 0 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb5da5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb5da5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.194+0000 7f5fb5da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 -- 172.21.15.163:0/2783876786 >> v2:172.21.15.70:3300/0 conn(0x7f5fb013e630 msgr2=0x7f5fb0071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 --2- 172.21.15.163:0/2783876786 >> v2:172.21.15.70:3300/0 conn(0x7f5fb013e630 0x7f5fb0071280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 -- 172.21.15.163:0/2783876786 >> v1:172.21.15.70:6789/0 conn(0x7f5fb0141940 legacy=0x7f5fb013df20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 monclient: found mon.noname-c 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 -- 172.21.15.163:0/2783876786 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb0073880 con 0x7f5fb00b28f0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb55a4640 1 --2- 172.21.15.163:0/2783876786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb00b28f0 0x7f5fb0070d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0076b90 tx=0x7f5fa402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab39d8d818b1f343 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 1 -- 172.21.15.163:0/2783876786 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4037070 con 0x7f5fb00b28f0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: dump: 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.207 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb7808640 5 monclient: authenticate success, global_id 4111 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 1 -- 172.21.15.163:0/2783876786 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa402ae20 con 0x7f5fb00b28f0 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb7808640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:35.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb7808640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fb7808640 4 set_mon_vals no callback set 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 1 -- 172.21.15.163:0/2783876786 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4032660 con 0x7f5fb00b28f0 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.195+0000 7f5fa3fff640 10 monclient: dump: 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.209 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 10 monclient: shutdown 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 -- 172.21.15.163:0/2783876786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb00b28f0 msgr2=0x7f5fb0070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 --2- 172.21.15.163:0/2783876786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb00b28f0 0x7f5fb0070d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0076b90 tx=0x7f5fa402a110 comp rx=0 tx=0).stop 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 -- 172.21.15.163:0/2783876786 shutdown_connections 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 --2- 172.21.15.163:0/2783876786 >> v2:172.21.15.70:3300/0 conn(0x7f5fb013e630 0x7f5fb0071280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 --2- 172.21.15.163:0/2783876786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb00b28f0 0x7f5fb0070d40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 -- 172.21.15.163:0/2783876786 >> 172.21.15.163:0/2783876786 conn(0x7f5fb00b3770 msgr2=0x7f5fb00b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 -- 172.21.15.163:0/2783876786 shutdown_connections 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.196+0000 7f5fb7808640 1 -- 172.21.15.163:0/2783876786 wait complete. 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 10 monclient: monmap: 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:35.211 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 librados: starting msgr at 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 librados: starting objecter 2024-09-16T03:49:35.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 Processor -- start 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 -- start start 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 librados: setting wanted keys 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 1 librados: calling monclient init 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.197+0000 7f5fb7808640 10 monclient: init 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5fb00b28f0 0x7f5fb0147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb00b28f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb013e630 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:35.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0141940 0x7f5fb0145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb0141940 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00b32b0 con 0x7f5fb0141940 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00b3130 con 0x7f5fb013e630 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb014b5d0 con 0x7f5fb00b28f0 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient(hunting): authenticate will time out at 750.766201s 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39738/0 (socket says 172.21.15.163:39738) 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 -- 172.21.15.163:0/1248933653 learned_addr learned my addr 172.21.15.163:0/1248933653 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): get_auth_request con 0x7f5fb013e630 auth_method 0 2024-09-16T03:49:35.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0141940 0x7f5fb0145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): get_auth_request con 0x7f5fb0141940 auth_method 0 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb5da5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 -- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5fb00b28f0 msgr2=0x7f5fb0147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5fb00b28f0 0x7f5fb0147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 -- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0141940 msgr2=0x7f5fb0145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0141940 0x7f5fb0145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 monclient: found mon.noname-b 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 -- 172.21.15.163:0/1248933653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb0149e20 con 0x7f5fb013e630 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb4da3640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00b3f0 tx=0x7f5fac00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f5fb013e630 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac015020 con 0x7f5fb013e630 2024-09-16T03:49:35.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: dump: 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fac011830 con 0x7f5fb013e630 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 5 monclient: authenticate success, global_id 4114 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac011c00 con 0x7f5fb013e630 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient: _renew_subs 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb014a100 con 0x7f5fb013e630 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa37fe640 4 set_mon_vals no callback set 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.198+0000 7f5fa1ffb640 10 monclient: dump: 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.218 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5fac018020 con 0x7f5fb013e630 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fb7808640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:35.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fb7808640 10 monclient: _renew_subs 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fb7808640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb01485f0 con 0x7f5fb013e630 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fb7808640 1 librados: init done 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f5fac011da0 con 0x7f5fb013e630 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 10 client.4114.objecter ms_dispatch 0x7f5fb0000b90 osd_map(2..2 src has 1..2) 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.199+0000 7f5fa1ffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.200+0000 7f5fb7808640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.200+0000 7f5fb7808640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.200+0000 7f5fb7808640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.200+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb01488b0 con 0x7f5fb013e630 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.202+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5fac01f020 con 0x7f5fb013e630 2024-09-16T03:49:35.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.202+0000 7f5fa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.202+0000 7f5fa1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:35.259 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:35.257+0000 7f14277b9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T03:49:35.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.272+0000 7f5fb7808640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0e4116b6-869c-4008-a494-02c6bf40f9a1", "id": 0}] 2024-09-16T03:49:35.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.272+0000 7f5fb7808640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0e4116b6-869c-4008-a494-02c6bf40f9a1", "id": 0}] 2024-09-16T03:49:35.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.272+0000 7f5fb7808640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.272+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0e4116b6-869c-4008-a494-02c6bf40f9a1", "id": 0} v 0) -- 0x7f5fb014a490 con 0x7f5fb013e630 2024-09-16T03:49:35.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.276+0000 7f5fa1ffb640 1 -- 172.21.15.163:0/1248933653 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0e4116b6-869c-4008-a494-02c6bf40f9a1", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f5fac01a9c0 con 0x7f5fb013e630 2024-09-16T03:49:35.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.276+0000 7f5fa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0e4116b6-869c-4008-a494-02c6bf40f9a1", "id": 0}] 2024-09-16T03:49:35.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.276+0000 7f5fa1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:35.278 INFO:teuthology.orchestra.run.smithi163.stdout:0 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 10 librados: watch_flush enter 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 10 librados: watch_flush exit 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 10 client.4114.objecter successfully canceled tick 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 10 monclient: shutdown 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 msgr2=0x7f5fb0148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00b3f0 tx=0x7f5fac00b8b0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 shutdown_connections 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0141940 0x7f5fb0145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fb013e630 0x7f5fb0148fa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 --2- 172.21.15.163:0/1248933653 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5fb00b28f0 0x7f5fb0147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.277+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 >> 172.21.15.163:0/1248933653 conn(0x7f5fb00b3770 msgr2=0x7f5fb006aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.278+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 shutdown_connections 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.278+0000 7f5fb7808640 1 -- 172.21.15.163:0/1248933653 wait complete. 2024-09-16T03:49:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.278+0000 7f5fb7808640 1 librados: shutdown 2024-09-16T03:49:35.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new b9adee7c-8804-49bc-894f-8f7461210815 1 2024-09-16T03:49:35.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 10 monclient: get_monmap_and_config 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 10 monclient: monmap: 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:35.386788+0000 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:35.386788+0000 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:35.388 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 1 Processor -- start 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 1 -- start start 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.386+0000 7f9581df8640 10 monclient: init 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f957c0b5650 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c146500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-b con 0x7f957c0b3120 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f957c146a40 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:35.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f957c0aae40 con 0x7f957c146a40 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f957c140b20 con 0x7f957c0b5650 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f957c0b5a30 con 0x7f957c0b3120 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f9581df8640 10 monclient(hunting): authenticate will time out at 750.955576s 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957b7fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f957c0b5650 0x7f957c140410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43644/0 (socket says 172.21.15.163:43644) 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957b7fe640 1 -- 172.21.15.163:0/4262638301 learned_addr learned my addr 172.21.15.163:0/4262638301 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 1 --2- 172.21.15.163:0/4262638301 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): get_auth_request con 0x7f957c0b3120 auth_method 0 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699324850 0 0) 0x7f957c140b20 con 0x7f957c0b5650 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.387+0000 7f957a7fc640 10 monclient(hunting): my global_id is 4117 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9564003520 con 0x7f957c0b5650 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530752601 0 0) 0x7f957c0aae40 con 0x7f957c146a40 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 10 monclient(hunting): my global_id is 4119 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f957c140b20 con 0x7f957c146a40 2024-09-16T03:49:35.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 1 -- 172.21.15.163:0/4262638301 >> v1:172.21.15.163:6789/0 conn(0x7f957c0b5650 legacy=0x7f957c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 1 -- 172.21.15.163:0/4262638301 >> v1:172.21.15.70:6789/0 conn(0x7f957c146a40 legacy=0x7f957c14d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 1 monclient: found mon.noname-b 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 1 -- 172.21.15.163:0/4262638301 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c0b5cd0 con 0x7f957c0b3120 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.388+0000 7f957affd640 1 --2- 172.21.15.163:0/4262638301 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c146500 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097d0 tx=0x7f956c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5fce3b3a3a6e75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c037070 con 0x7f957c0b3120 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: dump: 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.400 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f956c032440 con 0x7f957c0b3120 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 5 monclient: authenticate success, global_id 4109 2024-09-16T03:49:35.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 1 -- 172.21.15.163:0/4262638301 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c032810 con 0x7f957c0b3120 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 4 set_mon_vals no callback set 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f957a7fc640 10 monclient: dump: 2024-09-16T03:49:35.402 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 10 monclient: shutdown 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 -- 172.21.15.163:0/4262638301 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 msgr2=0x7f957c146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 --2- 172.21.15.163:0/4262638301 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c146500 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097d0 tx=0x7f956c02a180 comp rx=0 tx=0).stop 2024-09-16T03:49:35.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 -- 172.21.15.163:0/4262638301 shutdown_connections 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 --2- 172.21.15.163:0/4262638301 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c146500 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 -- 172.21.15.163:0/4262638301 >> 172.21.15.163:0/4262638301 conn(0x7f957c0679c0 msgr2=0x7f957c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 -- 172.21.15.163:0/4262638301 shutdown_connections 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.389+0000 7f9581df8640 1 -- 172.21.15.163:0/4262638301 wait complete. 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 10 monclient: monmap: 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 librados: starting msgr at 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 librados: starting objecter 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 Processor -- start 2024-09-16T03:49:35.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 -- start start 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 librados: setting wanted keys 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 1 librados: calling monclient init 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.390+0000 7f9581df8640 10 monclient: init 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-c con 0x7f957c0b3120 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 0x7f957c1d8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-a con 0x7f957c0b5650 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): picked mon.noname-b con 0x7f957c146a40 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f957c147030 con 0x7f957c0b5650 2024-09-16T03:49:35.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f957c146eb0 con 0x7f957c146a40 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f957c1471b0 con 0x7f957c0b3120 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f9581df8640 10 monclient(hunting): authenticate will time out at 750.959572s 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39752/0 (socket says 172.21.15.163:39752) 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 1 -- 172.21.15.163:0/3877437935 learned_addr learned my addr 172.21.15.163:0/3877437935 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): get_auth_request con 0x7f957c146a40 auth_method 0 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 0x7f957c1d8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): get_auth_request con 0x7f957c0b3120 auth_method 0 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): get_auth_request con 0x7f957c0b5650 auth_method 0 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.391+0000 7f957affd640 10 monclient(hunting): handle_auth_done global_id 4122 payload 824 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 -- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 msgr2=0x7f957c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 -- 172.21.15.163:0/3877437935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 msgr2=0x7f957c1d9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 monclient: found mon.noname-a 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 -- 172.21.15.163:0/3877437935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c0ae240 con 0x7f957c0b5650 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957b7fe640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957bfff640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f957affd640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 0x7f957c1d8b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f957c0b9e60 tx=0x7f956c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 client.?.objecter ms_handle_connect 0x7f957c0b5650 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c041030 con 0x7f957c0b5650 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: dump: 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.410 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f956c03d4e0 con 0x7f957c0b5650 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 5 monclient: authenticate success, global_id 4122 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 10 monclient: _renew_subs 2024-09-16T03:49:35.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957c1d85f0 con 0x7f957c0b5650 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c037040 con 0x7f957c0b5650 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 monclient: dump: 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.412 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 10 monclient: _renew_subs 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957c140e00 con 0x7f957c0b5650 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9581df8640 1 librados: init done 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9579ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f956c03d8f0 con 0x7f957c0b5650 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f956c0469f0 con 0x7f957c0b5650 2024-09-16T03:49:35.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 10 client.4122.objecter ms_dispatch 0x7f957c000b90 osd_map(3..3 src has 1..3) 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 3 client.4122.objecter handle_osd_map got epochs [3,3] > 0 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 3 client.4122.objecter handle_osd_map decoding full epoch 3 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.392+0000 7f9563fff640 20 client.4122.objecter dump_active .. 0 homeless 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.393+0000 7f9581df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.393+0000 7f9581df8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.393+0000 7f9581df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.393+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c146500 con 0x7f957c0b5650 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.395+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f956c04b020 con 0x7f957c0b5650 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.395+0000 7f9563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.395+0000 7f9563fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:35.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.465+0000 7f9581df8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b9adee7c-8804-49bc-894f-8f7461210815", "id": 1}] 2024-09-16T03:49:35.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.465+0000 7f9581df8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b9adee7c-8804-49bc-894f-8f7461210815", "id": 1}] 2024-09-16T03:49:35.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.465+0000 7f9581df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:35.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.465+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b9adee7c-8804-49bc-894f-8f7461210815", "id": 1} v 0) -- 0x7f957c1d87c0 con 0x7f957c0b5650 2024-09-16T03:49:35.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.466+0000 7f9563fff640 1 -- 172.21.15.163:0/3877437935 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9adee7c-8804-49bc-894f-8f7461210815", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f956c03dcd0 con 0x7f957c0b5650 2024-09-16T03:49:35.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.466+0000 7f9563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b9adee7c-8804-49bc-894f-8f7461210815", "id": 1}] 2024-09-16T03:49:35.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.466+0000 7f9563fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:35.468 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-09-16T03:49:35.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 10 librados: watch_flush enter 2024-09-16T03:49:35.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 10 librados: watch_flush exit 2024-09-16T03:49:35.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 20 client.4122.objecter shutdown clearing up homeless session... 2024-09-16T03:49:35.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 10 client.4122.objecter successfully canceled tick 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 10 monclient: shutdown 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 msgr2=0x7f957c1d8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 0x7f957c1d8b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f957c0b9e60 tx=0x7f956c0046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 shutdown_connections 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f957c146a40 0x7f957c1d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f957c0b5650 0x7f957c1d8b10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 --2- 172.21.15.163:0/3877437935 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f957c0b3120 0x7f957c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 >> 172.21.15.163:0/3877437935 conn(0x7f957c0679c0 msgr2=0x7f957c067da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 shutdown_connections 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 -- 172.21.15.163:0/3877437935 wait complete. 2024-09-16T03:49:35.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.467+0000 7f9581df8640 1 librados: shutdown 2024-09-16T03:49:35.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 9d85899b-73b5-4c63-9962-f74c96a2d208 2 2024-09-16T03:49:35.516 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:35.513+0000 7f37f6412040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T03:49:35.572 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:35.570+0000 7f37f6412040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T03:49:35.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.576+0000 7f520da77640 10 monclient: get_monmap_and_config 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.576+0000 7f520da77640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.576+0000 7f520da77640 10 monclient: monmap: 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:35.577095+0000 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:35.577095+0000 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:35.579 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:35.579 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:35.579 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:35.579 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 Processor -- start 2024-09-16T03:49:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 -- start start 2024-09-16T03:49:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient: init 2024-09-16T03:49:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f52080b5650 0x7f52080b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): picked mon.noname-b con 0x7f52080b5650 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5208140410 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5208140830 0x7f5208146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): picked mon.noname-c con 0x7f5208140830 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52080aae40 con 0x7f5208140410 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f52080b5a30 con 0x7f5208140830 2024-09-16T03:49:35.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f5208147150 con 0x7f52080b5650 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.577+0000 7f520da77640 10 monclient(hunting): authenticate will time out at 751.145910s 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5208140830 0x7f5208146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5208140830 0x7f5208146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39754/0 (socket says 172.21.15.163:39754) 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 -- 172.21.15.163:0/795448280 learned_addr learned my addr 172.21.15.163:0/795448280 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): get_auth_request con 0x7f5208140830 auth_method 0 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f52080b5650 0x7f52080b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): get_auth_request con 0x7f52080b5650 auth_method 0 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 -- 172.21.15.163:0/795448280 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f52080b5650 msgr2=0x7f52080b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f52080b5650 0x7f52080b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 -- 172.21.15.163:0/795448280 >> v1:172.21.15.163:6789/0 conn(0x7f5208140410 legacy=0x7f5208146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 monclient: found mon.noname-c 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.578+0000 7f5207fff640 1 -- 172.21.15.163:0/795448280 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52080b5cd0 con 0x7f5208140830 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f5207fff640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f5208146c10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5200009e50 tx=0x7f520002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f048085a512cebd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 1 -- 172.21.15.163:0/795448280 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200036070 con 0x7f5208140830 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: dump: 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.595 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 1 -- 172.21.15.163:0/795448280 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5200031600 con 0x7f5208140830 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f520da77640 5 monclient: authenticate success, global_id 4123 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f520da77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 1 -- 172.21.15.163:0/795448280 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52000319d0 con 0x7f5208140830 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f520da77640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:35.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f520da77640 4 set_mon_vals no callback set 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f52067fc640 10 monclient: dump: 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.597 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.579+0000 7f520da77640 10 monclient: shutdown 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 -- 172.21.15.163:0/795448280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 msgr2=0x7f5208146c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f5208146c10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5200009e50 tx=0x7f520002a680 comp rx=0 tx=0).stop 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 -- 172.21.15.163:0/795448280 shutdown_connections 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f5208146c10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 --2- 172.21.15.163:0/795448280 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f52080b5650 0x7f52080b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 -- 172.21.15.163:0/795448280 >> 172.21.15.163:0/795448280 conn(0x7f52080679c0 msgr2=0x7f5208067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 -- 172.21.15.163:0/795448280 shutdown_connections 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.580+0000 7f520da77640 1 -- 172.21.15.163:0/795448280 wait complete. 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 10 monclient: monmap: 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:35.599 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 librados: starting msgr at 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 librados: starting objecter 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 Processor -- start 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 -- start start 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 librados: setting wanted keys 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 1 librados: calling monclient init 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.581+0000 7f520da77640 10 monclient: init 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52080b5650 0x7f520814ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): picked mon.noname-a con 0x7f52080b5650 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:35.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5208140410 0x7f520814a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): picked mon.noname-c con 0x7f5208140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): picked mon.noname-b con 0x7f5208140830 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f520814da10 con 0x7f52080b5650 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f520814d890 con 0x7f5208140830 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f520814db90 con 0x7f5208140410 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f520da77640 10 monclient(hunting): authenticate will time out at 751.150866s 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f5207fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.582+0000 7f5207fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39762/0 (socket says 172.21.15.163:39762) 2024-09-16T03:49:35.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 -- 172.21.15.163:0/2883709310 learned_addr learned my addr 172.21.15.163:0/2883709310 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): get_auth_request con 0x7f5208140830 auth_method 0 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52080b5650 0x7f520814ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5208140410 0x7f520814a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): get_auth_request con 0x7f52080b5650 auth_method 0 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): get_auth_request con 0x7f5208140410 auth_method 0 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f52077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 -- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5208140410 msgr2=0x7f520814a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5208140410 0x7f520814a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 -- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52080b5650 msgr2=0x7f520814ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52080b5650 0x7f520814ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 monclient: found mon.noname-b 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 -- 172.21.15.163:0/2883709310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520814c020 con 0x7f5208140830 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5207fff640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5200009e50 tx=0x7f5200004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 client.?.objecter ms_handle_connect 0x7f5208140830 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200036050 con 0x7f5208140830 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: dump: 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.605 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5200042070 con 0x7f5208140830 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 5 monclient: authenticate success, global_id 4126 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200009510 con 0x7f5208140830 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 10 monclient: _renew_subs 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520814c590 con 0x7f5208140830 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 monclient: dump: 2024-09-16T03:49:35.606 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f5205ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:35.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 10 monclient: _renew_subs 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520813c6f0 con 0x7f5208140830 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f520da77640 1 librados: init done 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5200007bf0 con 0x7f5208140830 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f520003d450 con 0x7f5208140830 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 10 client.4126.objecter ms_dispatch 0x7f5208000b90 osd_map(4..4 src has 1..4) 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-09-16T03:49:35.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.583+0000 7f51effff640 20 client.4126.objecter dump_active .. 0 homeless 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.584+0000 7f520da77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.584+0000 7f520da77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.584+0000 7f520da77640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.584+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52080b3120 con 0x7f5208140830 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.586+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5200047460 con 0x7f5208140830 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.586+0000 7f51effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.586+0000 7f51effff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:35.610 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:35.605+0000 7f14277b9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T03:49:35.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.655+0000 7f520da77640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9d85899b-73b5-4c63-9962-f74c96a2d208", "id": 2}] 2024-09-16T03:49:35.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.655+0000 7f520da77640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9d85899b-73b5-4c63-9962-f74c96a2d208", "id": 2}] 2024-09-16T03:49:35.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.655+0000 7f520da77640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.655+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9d85899b-73b5-4c63-9962-f74c96a2d208", "id": 2} v 0) -- 0x7f520814d5f0 con 0x7f5208140830 2024-09-16T03:49:35.658 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:35.656+0000 7f14277b9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T03:49:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.660+0000 7f51effff640 1 -- 172.21.15.163:0/2883709310 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d85899b-73b5-4c63-9962-f74c96a2d208", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f5200047a50 con 0x7f5208140830 2024-09-16T03:49:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.660+0000 7f51effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9d85899b-73b5-4c63-9962-f74c96a2d208", "id": 2}] 2024-09-16T03:49:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.660+0000 7f51effff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:35.662 INFO:teuthology.orchestra.run.smithi163.stdout:2 2024-09-16T03:49:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 10 librados: watch_flush enter 2024-09-16T03:49:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 10 librados: watch_flush exit 2024-09-16T03:49:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-16T03:49:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 10 client.4126.objecter successfully canceled tick 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 10 monclient: shutdown 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 msgr2=0x7f520813c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5200009e50 tx=0x7f5200004a40 comp rx=0 tx=0).stop 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 shutdown_connections 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5208140830 0x7f520813c1b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5208140410 0x7f520814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 --2- 172.21.15.163:0/2883709310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52080b5650 0x7f520814ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 >> 172.21.15.163:0/2883709310 conn(0x7f52080679c0 msgr2=0x7f52080b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 shutdown_connections 2024-09-16T03:49:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 -- 172.21.15.163:0/2883709310 wait complete. 2024-09-16T03:49:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.661+0000 7f520da77640 1 librados: shutdown 2024-09-16T03:49:35.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new d31b8445-ae1e-4bfb-9588-5af8c9daf748 3 2024-09-16T03:49:35.724 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:35.721+0000 7f37f6412040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.768+0000 7f49f1c23640 10 monclient: get_monmap_and_config 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.768+0000 7f49f1c23640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.768+0000 7f49f1c23640 10 monclient: monmap: 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.770 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:35.769431+0000 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:35.769431+0000 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:35.771 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.769+0000 7f49f1c23640 1 Processor -- start 2024-09-16T03:49:35.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.769+0000 7f49f1c23640 1 -- start start 2024-09-16T03:49:35.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.769+0000 7f49f1c23640 10 monclient: init 2024-09-16T03:49:35.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f49ec0b5650 0x7f49ec0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-c con 0x7f49ec0b5650 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec140410 0x7f49ec140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-b con 0x7f49ec140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f49ec146500 0x7f49ec146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-a con 0x7f49ec146500 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec0b5a30 con 0x7f49ec146500 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f49ec0aae40 con 0x7f49ec0b5650 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec0b5cd0 con 0x7f49ec140410 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49f1c23640 10 monclient(hunting): authenticate will time out at 751.338500s 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f49ec0b5650 0x7f49ec0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f49ec0b5650 0x7f49ec0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39772/0 (socket says 172.21.15.163:39772) 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 -- 172.21.15.163:0/4221338610 learned_addr learned my addr 172.21.15.163:0/4221338610 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): get_auth_request con 0x7f49ec0b5650 auth_method 0 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 1 --2- 172.21.15.163:0/4221338610 >> v2:172.21.15.70:3300/0 conn(0x7f49ec146500 0x7f49ec146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec140410 0x7f49ec140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): get_auth_request con 0x7f49ec146500 auth_method 0 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): get_auth_request con 0x7f49ec140410 auth_method 0 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 -- 172.21.15.163:0/4221338610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec140410 msgr2=0x7f49ec140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec140410 0x7f49ec140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 -- 172.21.15.163:0/4221338610 >> v2:172.21.15.70:3300/0 conn(0x7f49ec146500 msgr2=0x7f49ec146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/4221338610 >> v2:172.21.15.70:3300/0 conn(0x7f49ec146500 0x7f49ec146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.770+0000 7f49eb7fe640 1 monclient: found mon.noname-c 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eb7fe640 1 -- 172.21.15.163:0/4221338610 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0aa9f0 con 0x7f49ec0b5650 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec0b3120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0b6130 tx=0x7f49dc02a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd894bb68b715782 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 1 -- 172.21.15.163:0/4221338610 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc037070 con 0x7f49ec0b5650 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: dump: 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49f1c23640 5 monclient: authenticate success, global_id 4129 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 1 -- 172.21.15.163:0/4221338610 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc032440 con 0x7f49ec0b5650 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49f1c23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49f1c23640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 1 -- 172.21.15.163:0/4221338610 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc032810 con 0x7f49ec0b5650 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49f1c23640 4 set_mon_vals no callback set 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.771+0000 7f49eaffd640 10 monclient: dump: 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.786 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 10 monclient: shutdown 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 -- 172.21.15.163:0/4221338610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 msgr2=0x7f49ec0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec0b3120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0b6130 tx=0x7f49dc02a140 comp rx=0 tx=0).stop 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 -- 172.21.15.163:0/4221338610 shutdown_connections 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 --2- 172.21.15.163:0/4221338610 >> v2:172.21.15.70:3300/0 conn(0x7f49ec146500 0x7f49ec146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec140410 0x7f49ec140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 --2- 172.21.15.163:0/4221338610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec0b3120 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 -- 172.21.15.163:0/4221338610 >> 172.21.15.163:0/4221338610 conn(0x7f49ec0679c0 msgr2=0x7f49ec067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 -- 172.21.15.163:0/4221338610 shutdown_connections 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.772+0000 7f49f1c23640 1 -- 172.21.15.163:0/4221338610 wait complete. 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 10 monclient: monmap: 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:35.788 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 librados: starting msgr at 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 librados: starting objecter 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 Processor -- start 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 -- start start 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 librados: setting wanted keys 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 1 librados: calling monclient init 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.773+0000 7f49f1c23640 10 monclient: init 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-b con 0x7f49ec0b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec140410 0x7f49ec136ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-a con 0x7f49ec140410 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec146500 0x7f49ec1373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): picked mon.noname-c con 0x7f49ec146500 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec143000 con 0x7f49ec140410 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1470e0 con 0x7f49ec0b5650 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec143180 con 0x7f49ec146500 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient(hunting): authenticate will time out at 751.342253s 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39786/0 (socket says 172.21.15.163:39786) 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 -- 172.21.15.163:0/1032858953 learned_addr learned my addr 172.21.15.163:0/1032858953 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): get_auth_request con 0x7f49ec0b5650 auth_method 0 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec140410 0x7f49ec136ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): get_auth_request con 0x7f49ec140410 auth_method 0 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec146500 0x7f49ec1373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): get_auth_request con 0x7f49ec146500 auth_method 0 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 -- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec146500 msgr2=0x7f49ec1373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec146500 0x7f49ec1373e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 -- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec140410 msgr2=0x7f49ec136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec140410 0x7f49ec136ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 monclient: found mon.noname-b 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 -- 172.21.15.163:0/1032858953 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0b6990 con 0x7f49ec0b5650 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49eb7fe640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f49dc02a180 tx=0x7f49dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 client.?.objecter ms_handle_connect 0x7f49ec0b5650 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc041070 con 0x7f49ec0b5650 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:35.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: dump: 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49dc032e50 con 0x7f49ec0b5650 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc037040 con 0x7f49ec0b5650 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49e8ff9640 10 monclient: dump: 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.795 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 5 monclient: authenticate success, global_id 4132 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient: _renew_subs 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49ea7fc640 4 set_mon_vals no callback set 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.774+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec144780 con 0x7f49ec0b5650 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49f1c23640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49f1c23640 10 monclient: _renew_subs 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49f1c23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1ce470 con 0x7f49ec0b5650 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49f1c23640 1 librados: init done 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f49dc044070 con 0x7f49ec0b5650 2024-09-16T03:49:35.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f49dc046860 con 0x7f49ec0b5650 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 10 client.4132.objecter ms_dispatch 0x7f49ec000b90 osd_map(5..5 src has 1..5) 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.775+0000 7f49e8ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.776+0000 7f49f1c23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.776+0000 7f49f1c23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.776+0000 7f49f1c23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.776+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec140830 con 0x7f49ec0b5650 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.778+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f49dc046450 con 0x7f49ec0b5650 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.778+0000 7f49e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.778+0000 7f49e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:35.818 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:35.816+0000 7f14277b9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T03:49:35.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.847+0000 7f49f1c23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d31b8445-ae1e-4bfb-9588-5af8c9daf748", "id": 3}] 2024-09-16T03:49:35.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.847+0000 7f49f1c23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d31b8445-ae1e-4bfb-9588-5af8c9daf748", "id": 3}] 2024-09-16T03:49:35.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.847+0000 7f49f1c23640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.847+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d31b8445-ae1e-4bfb-9588-5af8c9daf748", "id": 3} v 0) -- 0x7f49ec1ce700 con 0x7f49ec0b5650 2024-09-16T03:49:35.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.854+0000 7f49e8ff9640 1 -- 172.21.15.163:0/1032858953 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d31b8445-ae1e-4bfb-9588-5af8c9daf748", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f49dc046b30 con 0x7f49ec0b5650 2024-09-16T03:49:35.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.854+0000 7f49e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d31b8445-ae1e-4bfb-9588-5af8c9daf748", "id": 3}] 2024-09-16T03:49:35.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.854+0000 7f49e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:35.856 INFO:teuthology.orchestra.run.smithi163.stdout:3 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.854+0000 7f49f1c23640 10 librados: watch_flush enter 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 10 librados: watch_flush exit 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 10 client.4132.objecter successfully canceled tick 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 10 monclient: shutdown 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 msgr2=0x7f49ec1cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f49dc02a180 tx=0x7f49dc0046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 shutdown_connections 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f49ec146500 0x7f49ec1373e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec140410 0x7f49ec136ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 --2- 172.21.15.163:0/1032858953 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49ec0b5650 0x7f49ec1cdf30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 >> 172.21.15.163:0/1032858953 conn(0x7f49ec0679c0 msgr2=0x7f49ec1424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 shutdown_connections 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 -- 172.21.15.163:0/1032858953 wait complete. 2024-09-16T03:49:35.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.855+0000 7f49f1c23640 1 librados: shutdown 2024-09-16T03:49:35.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 4a3eca33-e8df-4b98-9fba-9dd4a9298cc6 4 2024-09-16T03:49:35.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 10 monclient: get_monmap_and_config 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 10 monclient: monmap: 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:35.964673+0000 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:35.964673+0000 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:35.966 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:35.967 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:35.967 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 1 Processor -- start 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 1 -- start start 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.964+0000 7f6697ac0640 10 monclient: init 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 0x7f66900b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f66900b5650 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f6690140410 0x7f6690140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f6690140410 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f6690146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f6690146500 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f66900b5a30 con 0x7f66900b5650 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f66900aae40 con 0x7f6690140410 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f66900b5cd0 con 0x7f6690146500 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f6697ac0640 10 monclient(hunting): authenticate will time out at 751.533438s 2024-09-16T03:49:35.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f6690140410 0x7f6690140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f6690140410 0x7f6690140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39788/0 (socket says 172.21.15.163:39788) 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 -- 172.21.15.163:0/3520911979 learned_addr learned my addr 172.21.15.163:0/3520911979 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): get_auth_request con 0x7f6690140410 auth_method 0 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 1 --2- 172.21.15.163:0/3520911979 >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 0x7f66900b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669605d640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f6690146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 10 monclient(hunting): get_auth_request con 0x7f66900b5650 auth_method 0 2024-09-16T03:49:35.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669605d640 10 monclient(hunting): get_auth_request con 0x7f6690146500 auth_method 0 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669605d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669605d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669605d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 -- 172.21.15.163:0/3520911979 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 msgr2=0x7f6690146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f6690146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 -- 172.21.15.163:0/3520911979 >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 msgr2=0x7f66900b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 --2- 172.21.15.163:0/3520911979 >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 0x7f66900b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 monclient: found mon.noname-c 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669505b640 1 -- 172.21.15.163:0/3520911979 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66900aa9f0 con 0x7f6690140410 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.965+0000 7f669585c640 1 --2- 172.21.15.163:0/3520911979 >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 0x7f66900b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669505b640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6690140410 0x7f6690140860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f66900abf00 tx=0x7f668c02a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc5814fa245195b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 1 -- 172.21.15.163:0/3520911979 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c037070 con 0x7f6690140410 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: dump: 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.979 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f6697ac0640 5 monclient: authenticate success, global_id 4135 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f6697ac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:35.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 1 -- 172.21.15.163:0/3520911979 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c032440 con 0x7f6690140410 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f6697ac0640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f6697ac0640 4 set_mon_vals no callback set 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 1 -- 172.21.15.163:0/3520911979 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c032810 con 0x7f6690140410 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.966+0000 7f669485a640 10 monclient: dump: 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.981 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 10 monclient: shutdown 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 -- 172.21.15.163:0/3520911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6690140410 msgr2=0x7f6690140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6690140410 0x7f6690140860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f66900abf00 tx=0x7f668c02a140 comp rx=0 tx=0).stop 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 -- 172.21.15.163:0/3520911979 shutdown_connections 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f6690146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 --2- 172.21.15.163:0/3520911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6690140410 0x7f6690140860 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 --2- 172.21.15.163:0/3520911979 >> v2:172.21.15.70:3300/0 conn(0x7f66900b5650 0x7f66900b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 -- 172.21.15.163:0/3520911979 >> 172.21.15.163:0/3520911979 conn(0x7f66900679c0 msgr2=0x7f6690067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:35.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 -- 172.21.15.163:0/3520911979 shutdown_connections 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.967+0000 7f6697ac0640 1 -- 172.21.15.163:0/3520911979 wait complete. 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 10 monclient: build_initial_monmap 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 10 monclient: monmap: 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:35.983 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 librados: starting msgr at 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 librados: starting objecter 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 Processor -- start 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 -- start start 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 librados: setting wanted keys 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 1 librados: calling monclient init 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 10 monclient: init 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.968+0000 7f6697ac0640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-b con 0x7f66900b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6690140410 0x7f66901cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-a con 0x7f6690140410 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f66901ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): picked mon.noname-c con 0x7f6690146500 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6690143000 con 0x7f6690140410 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f66901470e0 con 0x7f66900b5650 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f6690143180 con 0x7f6690146500 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient(hunting): authenticate will time out at 751.537099s 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39800/0 (socket says 172.21.15.163:39800) 2024-09-16T03:49:35.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 -- 172.21.15.163:0/2197755835 learned_addr learned my addr 172.21.15.163:0/2197755835 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): get_auth_request con 0x7f66900b5650 auth_method 0 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6690140410 0x7f66901cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f66901ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): get_auth_request con 0x7f6690140410 auth_method 0 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): get_auth_request con 0x7f6690146500 auth_method 0 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:35.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669505b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669605d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 -- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 msgr2=0x7f66901ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f66901ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 -- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6690140410 msgr2=0x7f66901cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:35.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6690140410 0x7f66901cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 monclient: found mon.noname-b 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 -- 172.21.15.163:0/2197755835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66901cdf40 con 0x7f66900b5650 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f669585c640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f668000e7e0 tx=0x7f668000eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 client.?.objecter ms_handle_connect 0x7f66900b5650 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668000f040 con 0x7f66900b5650 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: dump: 2024-09-16T03:49:35.988 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6680004030 con 0x7f66900b5650 2024-09-16T03:49:35.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 5 monclient: authenticate success, global_id 4138 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient: _renew_subs 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66901ce220 con 0x7f66900b5650 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680004400 con 0x7f66900b5650 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f66867fc640 10 monclient: dump: 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:35.990 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6687fff640 4 set_mon_vals no callback set 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient: _renew_subs 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66901ce7e0 con 0x7f66900b5650 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.969+0000 7f6697ac0640 1 librados: init done 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f668000ca60 con 0x7f66900b5650 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6680013a60 con 0x7f66900b5650 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 10 client.4138.objecter ms_dispatch 0x7f6690000b90 osd_map(6..6 src has 1..6) 2024-09-16T03:49:35.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f66867fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f6697ac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f6697ac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f6697ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.970+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6690140840 con 0x7f66900b5650 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.972+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f6680017020 con 0x7f66900b5650 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.972+0000 7f66867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:35.972+0000 7f66867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:36.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.042+0000 7f6697ac0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4a3eca33-e8df-4b98-9fba-9dd4a9298cc6", "id": 4}] 2024-09-16T03:49:36.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.042+0000 7f6697ac0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4a3eca33-e8df-4b98-9fba-9dd4a9298cc6", "id": 4}] 2024-09-16T03:49:36.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.042+0000 7f6697ac0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.042+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a3eca33-e8df-4b98-9fba-9dd4a9298cc6", "id": 4} v 0) -- 0x7f6690072230 con 0x7f66900b5650 2024-09-16T03:49:36.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f66867fc640 1 -- 172.21.15.163:0/2197755835 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a3eca33-e8df-4b98-9fba-9dd4a9298cc6", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6680013d10 con 0x7f66900b5650 2024-09-16T03:49:36.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f66867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4a3eca33-e8df-4b98-9fba-9dd4a9298cc6", "id": 4}] 2024-09-16T03:49:36.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f66867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:36.050 INFO:teuthology.orchestra.run.smithi163.stdout:4 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f6697ac0640 10 librados: watch_flush enter 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f6697ac0640 10 librados: watch_flush exit 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f6697ac0640 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.048+0000 7f6697ac0640 10 client.4138.objecter successfully canceled tick 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 10 monclient: shutdown 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 msgr2=0x7f66901cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f668000e7e0 tx=0x7f668000eca0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 shutdown_connections 2024-09-16T03:49:36.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f6690146500 0x7f66901ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6690140410 0x7f66901cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 --2- 172.21.15.163:0/2197755835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66900b5650 0x7f66901cef70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 >> 172.21.15.163:0/2197755835 conn(0x7f66900679c0 msgr2=0x7f6690141e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 shutdown_connections 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 -- 172.21.15.163:0/2197755835 wait complete. 2024-09-16T03:49:36.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.049+0000 7f6697ac0640 1 librados: shutdown 2024-09-16T03:49:36.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 3da90f35-297c-4a03-921c-428685f5e5a6 5 2024-09-16T03:49:36.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.159+0000 7f1d53cf1640 10 monclient: get_monmap_and_config 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.159+0000 7f1d53cf1640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.159+0000 7f1d53cf1640 10 monclient: monmap: 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:36.159766+0000 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:36.159766+0000 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:36.161 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:36.162 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.159+0000 7f1d53cf1640 1 Processor -- start 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 -- start start 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient: init 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1d4c0b28f0 0x7f1d4c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f1d4c0b28f0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f1d4c141940 0x7f1d4c071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f1d4c141940 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d4c0b30f0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d4c1411a0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1d4c0b2cd0 con 0x7f1d4c0b28f0 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f1d4c0735a0 con 0x7f1d4c141940 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d53cf1640 10 monclient(hunting): authenticate will time out at 751.728701s 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f1d4c141940 0x7f1d4c071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f1d4c141940 0x7f1d4c071280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39812/0 (socket says 172.21.15.163:39812) 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 1 -- 172.21.15.163:0/1464203671 learned_addr learned my addr 172.21.15.163:0/1464203671 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): get_auth_request con 0x7f1d4c141940 auth_method 0 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d51a8d640 1 --2- 172.21.15.163:0/1464203671 >> v2:172.21.15.70:3300/0 conn(0x7f1d4c0b28f0 0x7f1d4c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d51a8d640 10 monclient(hunting): get_auth_request con 0x7f1d4c0b28f0 auth_method 0 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d51a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d51a8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.160+0000 7f1d51a8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 -- 172.21.15.163:0/1464203671 >> v2:172.21.15.70:3300/0 conn(0x7f1d4c0b28f0 msgr2=0x7f1d4c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 --2- 172.21.15.163:0/1464203671 >> v2:172.21.15.70:3300/0 conn(0x7f1d4c0b28f0 0x7f1d4c070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 -- 172.21.15.163:0/1464203671 >> v1:172.21.15.70:6789/0 conn(0x7f1d4c0b30f0 legacy=0x7f1d4c13df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 monclient: found mon.noname-c 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 -- 172.21.15.163:0/1464203671 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4c073880 con 0x7f1d4c141940 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d5128c640 1 --2- 172.21.15.163:0/1464203671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c141940 0x7f1d4c071280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c074380 tx=0x7f1d3c02a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=584bf0a8621b2b60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.161+0000 7f1d50a8b640 1 -- 172.21.15.163:0/1464203671 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c037070 con 0x7f1d4c141940 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: dump: 2024-09-16T03:49:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 1 -- 172.21.15.163:0/1464203671 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3c032440 con 0x7f1d4c141940 2024-09-16T03:49:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 5 monclient: authenticate success, global_id 4141 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 1 -- 172.21.15.163:0/1464203671 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c032810 con 0x7f1d4c141940 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 4 set_mon_vals no callback set 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d50a8b640 10 monclient: dump: 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.176 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 10 monclient: shutdown 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1464203671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c141940 msgr2=0x7f1d4c071280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1464203671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c141940 0x7f1d4c071280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c074380 tx=0x7f1d3c02a150 comp rx=0 tx=0).stop 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1464203671 shutdown_connections 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1464203671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c141940 0x7f1d4c071280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1464203671 >> v2:172.21.15.70:3300/0 conn(0x7f1d4c0b28f0 0x7f1d4c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1464203671 >> 172.21.15.163:0/1464203671 conn(0x7f1d4c0b3770 msgr2=0x7f1d4c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1464203671 shutdown_connections 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.162+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1464203671 wait complete. 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 10 monclient: monmap: 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 librados: starting msgr at 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 librados: starting objecter 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 Processor -- start 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 -- start start 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 librados: setting wanted keys 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 1 librados: calling monclient init 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.163+0000 7f1d53cf1640 10 monclient: init 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1d4c0b28f0 0x7f1d4c14adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f1d4c0b28f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f1d4c0b30f0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:36.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d4c141940 0x7f1d4c1dbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f1d4c141940 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c13e7f0 con 0x7f1d4c141940 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c13e670 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f1d4c14b5d0 con 0x7f1d4c0b28f0 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d53cf1640 10 monclient(hunting): authenticate will time out at 751.732516s 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39816/0 (socket says 172.21.15.163:39816) 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 -- 172.21.15.163:0/1636889864 learned_addr learned my addr 172.21.15.163:0/1636889864 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:36.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): get_auth_request con 0x7f1d4c0b30f0 auth_method 0 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d4c141940 0x7f1d4c1dbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1d4c0b28f0 0x7f1d4c14adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): get_auth_request con 0x7f1d4c141940 auth_method 0 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): get_auth_request con 0x7f1d4c0b28f0 auth_method 0 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5228e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d51a8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 -- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1d4c0b28f0 msgr2=0x7f1d4c14adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1d4c0b28f0 0x7f1d4c14adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 -- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d4c141940 msgr2=0x7f1d4c1dbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d4c141940 0x7f1d4c1dbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 monclient: found mon.noname-b 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.164+0000 7f1d5128c640 1 -- 172.21.15.163:0/1636889864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4c0ac6c0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d5128c640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c02a190 tx=0x7f1d3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 client.?.objecter ms_handle_connect 0x7f1d4c0b30f0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c037040 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: dump: 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.183 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3c042070 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 5 monclient: authenticate success, global_id 4144 2024-09-16T03:49:36.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 10 monclient: _renew_subs 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d4c14b3f0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c030880 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 monclient: dump: 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.185 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 10 monclient: _renew_subs 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d4c0ab770 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d53cf1640 1 librados: init done 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d43fff640 4 set_mon_vals no callback set 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1d3c030a20 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f1d3c047af0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 10 client.4144.objecter ms_dispatch 0x7f1d4c000b90 osd_map(7..7 src has 1..7) 2024-09-16T03:49:36.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.165+0000 7f1d427fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.166+0000 7f1d53cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.166+0000 7f1d53cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.166+0000 7f1d53cf1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.166+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d4c13df20 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.168+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1d4c14b3f0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.168+0000 7f1d427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.168+0000 7f1d427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.252+0000 7f1d53cf1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3da90f35-297c-4a03-921c-428685f5e5a6", "id": 5}] 2024-09-16T03:49:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.252+0000 7f1d53cf1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3da90f35-297c-4a03-921c-428685f5e5a6", "id": 5}] 2024-09-16T03:49:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.252+0000 7f1d53cf1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.252+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3da90f35-297c-4a03-921c-428685f5e5a6", "id": 5} v 0) -- 0x7f1d4c072fe0 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.257+0000 7f1d427fc640 1 -- 172.21.15.163:0/1636889864 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3da90f35-297c-4a03-921c-428685f5e5a6", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f1d3c047460 con 0x7f1d4c0b30f0 2024-09-16T03:49:36.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.257+0000 7f1d427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3da90f35-297c-4a03-921c-428685f5e5a6", "id": 5}] 2024-09-16T03:49:36.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.257+0000 7f1d427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:36.259 INFO:teuthology.orchestra.run.smithi163.stdout:5 2024-09-16T03:49:36.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 10 librados: watch_flush enter 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 10 librados: watch_flush exit 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 20 client.4144.objecter shutdown clearing up homeless session... 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 10 client.4144.objecter successfully canceled tick 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 10 monclient: shutdown 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 msgr2=0x7f1d4c1dba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c02a190 tx=0x7f1d3c0046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 shutdown_connections 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d4c141940 0x7f1d4c1dbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1d4c0b30f0 0x7f1d4c1dba20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 --2- 172.21.15.163:0/1636889864 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1d4c0b28f0 0x7f1d4c14adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 >> 172.21.15.163:0/1636889864 conn(0x7f1d4c0b3770 msgr2=0x7f1d4c13ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 shutdown_connections 2024-09-16T03:49:36.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 -- 172.21.15.163:0/1636889864 wait complete. 2024-09-16T03:49:36.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.258+0000 7f1d53cf1640 1 librados: shutdown 2024-09-16T03:49:36.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 0008e277-c3bd-45c8-a0f6-82494d859a05 6 2024-09-16T03:49:36.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 10 monclient: get_monmap_and_config 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 10 monclient: monmap: 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:36.368770+0000 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:36.368770+0000 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:36.370 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:36.371 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 1 Processor -- start 2024-09-16T03:49:36.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 1 -- start start 2024-09-16T03:49:36.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.368+0000 7f3d62f83640 10 monclient: init 2024-09-16T03:49:36.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d5c0b5470 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d5c13cbd0 0x7f3d5c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c13cbd0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f3d5c143040 0x7f3d5c143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c143040 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5c0ac040 con 0x7f3d5c0b5470 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5c0b5f60 con 0x7f3d5c13cbd0 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5c0ac2b0 con 0x7f3d5c143040 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d62f83640 10 monclient(hunting): authenticate will time out at 751.937642s 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d60d1f640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3d5c0b5470 0x7f3d5c0b5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43660/0 (socket says 172.21.15.163:43660) 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d60d1f640 1 -- 172.21.15.163:0/2693877331 learned_addr learned my addr 172.21.15.163:0/2693877331 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 1 --2- 172.21.15.163:0/2693877331 >> v2:172.21.15.163:3300/0 conn(0x7f3d5c143040 0x7f3d5c143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): get_auth_request con 0x7f3d5c143040 auth_method 0 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d5bfff640 1 --2- 172.21.15.163:0/2693877331 >> v2:172.21.15.70:3300/0 conn(0x7f3d5c13cbd0 0x7f3d5c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request con 0x7f3d5c13cbd0 auth_method 0 2024-09-16T03:49:36.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.369+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 -- 172.21.15.163:0/2693877331 >> v2:172.21.15.70:3300/0 conn(0x7f3d5c13cbd0 msgr2=0x7f3d5c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 --2- 172.21.15.163:0/2693877331 >> v2:172.21.15.70:3300/0 conn(0x7f3d5c13cbd0 0x7f3d5c13d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 -- 172.21.15.163:0/2693877331 >> v1:172.21.15.163:6789/0 conn(0x7f3d5c0b5470 legacy=0x7f3d5c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 monclient: found mon.noname-c 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 -- 172.21.15.163:0/2693877331 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c0aaef0 con 0x7f3d5c143040 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d61520640 1 --2- 172.21.15.163:0/2693877331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c143040 0x7f3d5c143470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0b4b70 tx=0x7f3d5402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b66edcdb7729c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 1 -- 172.21.15.163:0/2693877331 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54007740 con 0x7f3d5c143040 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: dump: 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d62f83640 5 monclient: authenticate success, global_id 4147 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 1 -- 172.21.15.163:0/2693877331 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d540078e0 con 0x7f3d5c143040 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d62f83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d62f83640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d62f83640 4 set_mon_vals no callback set 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 1 -- 172.21.15.163:0/2693877331 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54007cb0 con 0x7f3d5c143040 2024-09-16T03:49:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.370+0000 7f3d5b7fe640 10 monclient: dump: 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 10 monclient: shutdown 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 -- 172.21.15.163:0/2693877331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c143040 msgr2=0x7f3d5c143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 --2- 172.21.15.163:0/2693877331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c143040 0x7f3d5c143470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0b4b70 tx=0x7f3d5402a680 comp rx=0 tx=0).stop 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 -- 172.21.15.163:0/2693877331 shutdown_connections 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 --2- 172.21.15.163:0/2693877331 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c143040 0x7f3d5c143470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 --2- 172.21.15.163:0/2693877331 >> v2:172.21.15.70:3300/0 conn(0x7f3d5c13cbd0 0x7f3d5c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 -- 172.21.15.163:0/2693877331 >> 172.21.15.163:0/2693877331 conn(0x7f3d5c0679c0 msgr2=0x7f3d5c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 -- 172.21.15.163:0/2693877331 shutdown_connections 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.371+0000 7f3d62f83640 1 -- 172.21.15.163:0/2693877331 wait complete. 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 10 monclient: monmap: 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 librados: starting msgr at 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 librados: starting objecter 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 Processor -- start 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 -- start start 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 librados: setting wanted keys 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 1 librados: calling monclient init 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.372+0000 7f3d62f83640 10 monclient: init 2024-09-16T03:49:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c0b5470 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d5c13cbd0 0x7f3d5c149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c13cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c143040 0x7f3d5c14bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c143040 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c143b30 con 0x7f3d5c143040 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1439b0 con 0x7f3d5c0b5470 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c143cb0 con 0x7f3d5c13cbd0 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d62f83640 10 monclient(hunting): authenticate will time out at 751.941308s 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39828/0 (socket says 172.21.15.163:39828) 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 -- 172.21.15.163:0/665873142 learned_addr learned my addr 172.21.15.163:0/665873142 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): get_auth_request con 0x7f3d5c0b5470 auth_method 0 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d5c13cbd0 0x7f3d5c149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c143040 0x7f3d5c14bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request con 0x7f3d5c13cbd0 auth_method 0 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): get_auth_request con 0x7f3d5c143040 auth_method 0 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d61520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 -- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d5c13cbd0 msgr2=0x7f3d5c149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d5c13cbd0 0x7f3d5c149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 -- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c143040 msgr2=0x7f3d5c14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c143040 0x7f3d5c14bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 monclient: found mon.noname-b 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 -- 172.21.15.163:0/665873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c14b220 con 0x7f3d5c0b5470 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d60d1f640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c430 tx=0x7f3d5000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d597fa640 10 client.?.objecter ms_handle_connect 0x7f3d5c0b5470 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.373+0000 7f3d597fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50017070 con 0x7f3d5c0b5470 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: dump: 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d50005ce0 con 0x7f3d5c0b5470 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 5 monclient: authenticate success, global_id 4153 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 10 monclient: _renew_subs 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c14b560 con 0x7f3d5c0b5470 2024-09-16T03:49:36.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500072f0 con 0x7f3d5c0b5470 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 monclient: dump: 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d5affd640 4 set_mon_vals no callback set 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 10 monclient: _renew_subs 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c149f20 con 0x7f3d5c0b5470 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d62f83640 1 librados: init done 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3d5001a070 con 0x7f3d5c0b5470 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f3d500129f0 con 0x7f3d5c0b5470 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 10 client.4153.objecter ms_dispatch 0x7f3d5c000b90 osd_map(8..8 src has 1..8) 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.374+0000 7f3d597fa640 20 client.4153.objecter dump_active .. 0 homeless 2024-09-16T03:49:36.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.375+0000 7f3d62f83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.375+0000 7f3d62f83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.375+0000 7f3d62f83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.375+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d5c13d020 con 0x7f3d5c0b5470 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.377+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3d50021020 con 0x7f3d5c0b5470 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.377+0000 7f3d597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.377+0000 7f3d597fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:36.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.448+0000 7f3d62f83640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0008e277-c3bd-45c8-a0f6-82494d859a05", "id": 6}] 2024-09-16T03:49:36.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.448+0000 7f3d62f83640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0008e277-c3bd-45c8-a0f6-82494d859a05", "id": 6}] 2024-09-16T03:49:36.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.448+0000 7f3d62f83640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.448+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0008e277-c3bd-45c8-a0f6-82494d859a05", "id": 6} v 0) -- 0x7f3d5c072230 con 0x7f3d5c0b5470 2024-09-16T03:49:36.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.452+0000 7f3d597fa640 1 -- 172.21.15.163:0/665873142 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0008e277-c3bd-45c8-a0f6-82494d859a05", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f3d5c14b560 con 0x7f3d5c0b5470 2024-09-16T03:49:36.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.452+0000 7f3d597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0008e277-c3bd-45c8-a0f6-82494d859a05", "id": 6}] 2024-09-16T03:49:36.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.452+0000 7f3d597fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:36.454 INFO:teuthology.orchestra.run.smithi163.stdout:6 2024-09-16T03:49:36.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 10 librados: watch_flush enter 2024-09-16T03:49:36.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 10 librados: watch_flush exit 2024-09-16T03:49:36.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 20 client.4153.objecter shutdown clearing up homeless session... 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 10 client.4153.objecter successfully canceled tick 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 10 monclient: shutdown 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 msgr2=0x7f3d5c13fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c430 tx=0x7f3d5000c8f0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 shutdown_connections 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c143040 0x7f3d5c14bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d5c13cbd0 0x7f3d5c149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 --2- 172.21.15.163:0/665873142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d5c0b5470 0x7f3d5c13fab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 >> 172.21.15.163:0/665873142 conn(0x7f3d5c0679c0 msgr2=0x7f3d5c1414f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 shutdown_connections 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 -- 172.21.15.163:0/665873142 wait complete. 2024-09-16T03:49:36.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.453+0000 7f3d62f83640 1 librados: shutdown 2024-09-16T03:49:36.465 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new e67e5a12-f19c-485d-8435-b280ab0335db 7 2024-09-16T03:49:36.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 10 monclient: get_monmap_and_config 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 10 monclient: monmap: 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:49:36.562869+0000 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:49:36.562869+0000 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:36.564 INFO:teuthology.orchestra.run.smithi163.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:36.565 INFO:teuthology.orchestra.run.smithi163.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:36.565 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 1 Processor -- start 2024-09-16T03:49:36.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 1 -- start start 2024-09-16T03:49:36.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.562+0000 7f07b88b7640 10 monclient: init 2024-09-16T03:49:36.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f07b40b5650 0x7f07b40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-c con 0x7f07b40b5650 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f07b4140410 0x7f07b4140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-a con 0x7f07b4140410 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4146500 0x7f07b4146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-b con 0x7f07b4146500 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f07b40b5a30 con 0x7f07b4140410 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f07b40aae40 con 0x7f07b40b5650 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f07b40b5cd0 con 0x7f07b4146500 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b88b7640 10 monclient(hunting): authenticate will time out at 752.131723s 2024-09-16T03:49:36.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f07b40b5650 0x7f07b40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f07b40b5650 0x7f07b40b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39840/0 (socket says 172.21.15.163:39840) 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 1 -- 172.21.15.163:0/3715799263 learned_addr learned my addr 172.21.15.163:0/3715799263 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 10 monclient(hunting): get_auth_request con 0x7f07b40b5650 auth_method 0 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b2d9d640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4146500 0x7f07b4146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b1d9b640 1 --2- 172.21.15.163:0/3715799263 >> v2:172.21.15.70:3300/0 conn(0x7f07b4140410 0x7f07b4140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b2d9d640 10 monclient(hunting): get_auth_request con 0x7f07b4146500 auth_method 0 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.563+0000 7f07b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): get_auth_request con 0x7f07b4140410 auth_method 0 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 -- 172.21.15.163:0/3715799263 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4146500 msgr2=0x7f07b4146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4146500 0x7f07b4146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 -- 172.21.15.163:0/3715799263 >> v2:172.21.15.70:3300/0 conn(0x7f07b4140410 msgr2=0x7f07b4140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 --2- 172.21.15.163:0/3715799263 >> v2:172.21.15.70:3300/0 conn(0x7f07b4140410 0x7f07b4140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 monclient: found mon.noname-c 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 -- 172.21.15.163:0/3715799263 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b40aa9f0 con 0x7f07b40b5650 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.564+0000 7f07b259c640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40b3120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07b40b6130 tx=0x7f07a802a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c882e63271a5c043 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 1 -- 172.21.15.163:0/3715799263 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8037070 con 0x7f07b40b5650 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: dump: 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.582 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 1 -- 172.21.15.163:0/3715799263 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a8032440 con 0x7f07b40b5650 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 5 monclient: authenticate success, global_id 4156 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 1 -- 172.21.15.163:0/3715799263 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8032810 con 0x7f07b40b5650 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b159a640 10 monclient: dump: 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.585 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 4 set_mon_vals no callback set 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 10 monclient: shutdown 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 -- 172.21.15.163:0/3715799263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 msgr2=0x7f07b40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40b3120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07b40b6130 tx=0x7f07a802a140 comp rx=0 tx=0).stop 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 -- 172.21.15.163:0/3715799263 shutdown_connections 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4146500 0x7f07b4146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 --2- 172.21.15.163:0/3715799263 >> v2:172.21.15.70:3300/0 conn(0x7f07b4140410 0x7f07b4140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 --2- 172.21.15.163:0/3715799263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40b3120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 -- 172.21.15.163:0/3715799263 >> 172.21.15.163:0/3715799263 conn(0x7f07b40679c0 msgr2=0x7f07b4067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 -- 172.21.15.163:0/3715799263 shutdown_connections 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.565+0000 7f07b88b7640 1 -- 172.21.15.163:0/3715799263 wait complete. 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.566+0000 7f07b88b7640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.566+0000 7f07b88b7640 10 monclient: monmap: 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.566+0000 7f07b88b7640 1 librados: starting msgr at 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.566+0000 7f07b88b7640 1 librados: starting objecter 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 Processor -- start 2024-09-16T03:49:36.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 -- start start 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 librados: setting wanted keys 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 librados: calling monclient init 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient: init 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-b con 0x7f07b40b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4140410 0x7f07b40ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-c con 0x7f07b4140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07b4146500 0x7f07b40b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): picked mon.noname-a con 0x7f07b4146500 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07b4143000 con 0x7f07b4146500 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07b41470e0 con 0x7f07b40b5650 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f07b4143180 con 0x7f07b4140410 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b88b7640 10 monclient(hunting): authenticate will time out at 752.135735s 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39854/0 (socket says 172.21.15.163:39854) 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 1 -- 172.21.15.163:0/2189409973 learned_addr learned my addr 172.21.15.163:0/2189409973 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): get_auth_request con 0x7f07b40b5650 auth_method 0 2024-09-16T03:49:36.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4140410 0x7f07b40ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): get_auth_request con 0x7f07b4140410 auth_method 0 2024-09-16T03:49:36.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b2d9d640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07b4146500 0x7f07b40b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b2d9d640 10 monclient(hunting): get_auth_request con 0x7f07b4146500 auth_method 0 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.567+0000 7f07b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 10 monclient(hunting): handle_auth_done global_id 4159 payload 824 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 -- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4140410 msgr2=0x7f07b40ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4140410 0x7f07b40ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 -- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07b4146500 msgr2=0x7f07b40b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07b4146500 0x7f07b40b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 monclient: found mon.noname-b 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 -- 172.21.15.163:0/2189409973 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b40af160 con 0x7f07b40b5650 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b259c640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 client.?.objecter ms_handle_connect 0x7f07b40b5650 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8041020 con 0x7f07b40b5650 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: dump: 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.595 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 5 monclient: authenticate success, global_id 4159 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a803d620 con 0x7f07b40b5650 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 10 monclient: _renew_subs 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b40acde0 con 0x7f07b40b5650 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8037040 con 0x7f07b40b5650 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 monclient: dump: 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 10 monclient: _renew_subs 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b0d99640 4 set_mon_vals no callback set 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b40b22d0 con 0x7f07b40b5650 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07b88b7640 1 librados: init done 2024-09-16T03:49:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f07a8044020 con 0x7f07b40b5650 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f07a8046c20 con 0x7f07b40b5650 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 10 client.4159.objecter ms_dispatch 0x7f07b4000b90 osd_map(9..9 src has 1..9) 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.568+0000 7f07a2ffd640 20 client.4159.objecter dump_active .. 0 homeless 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.569+0000 7f07b88b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.569+0000 7f07b88b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.569+0000 7f07b88b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.569+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b4140860 con 0x7f07b40b5650 2024-09-16T03:49:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.573+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f07a804b030 con 0x7f07b40b5650 2024-09-16T03:49:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.573+0000 7f07a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.573+0000 7f07a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:36.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.643+0000 7f07b88b7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e67e5a12-f19c-485d-8435-b280ab0335db", "id": 7}] 2024-09-16T03:49:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.643+0000 7f07b88b7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e67e5a12-f19c-485d-8435-b280ab0335db", "id": 7}] 2024-09-16T03:49:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.643+0000 7f07b88b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.643+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e67e5a12-f19c-485d-8435-b280ab0335db", "id": 7} v 0) -- 0x7f07b4072230 con 0x7f07b40b5650 2024-09-16T03:49:36.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.646+0000 7f07a2ffd640 1 -- 172.21.15.163:0/2189409973 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e67e5a12-f19c-485d-8435-b280ab0335db", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f07a8046400 con 0x7f07b40b5650 2024-09-16T03:49:36.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.646+0000 7f07a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e67e5a12-f19c-485d-8435-b280ab0335db", "id": 7}] 2024-09-16T03:49:36.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.646+0000 7f07a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:36.649 INFO:teuthology.orchestra.run.smithi163.stdout:7 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 10 librados: watch_flush enter 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 10 librados: watch_flush exit 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 20 client.4159.objecter shutdown clearing up homeless session... 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 10 client.4159.objecter successfully canceled tick 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 10 monclient: shutdown 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 msgr2=0x7f07b40adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a80046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 shutdown_connections 2024-09-16T03:49:36.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07b4146500 0x7f07b40b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f07b4140410 0x7f07b40ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 --2- 172.21.15.163:0/2189409973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07b40b5650 0x7f07b40adb60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 >> 172.21.15.163:0/2189409973 conn(0x7f07b40679c0 msgr2=0x7f07b41422a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 shutdown_connections 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 -- 172.21.15.163:0/2189409973 wait complete. 2024-09-16T03:49:36.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-09-16T03:49:36.647+0000 7f07b88b7640 1 librados: shutdown 2024-09-16T03:49:36.659 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T03:49:36.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T03:49:36.662 INFO:tasks.ceph.osd.0:Started 2024-09-16T03:49:36.662 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T03:49:36.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T03:49:36.664 INFO:tasks.ceph.osd.1:Started 2024-09-16T03:49:36.665 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T03:49:36.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T03:49:36.667 INFO:tasks.ceph.osd.2:Started 2024-09-16T03:49:36.667 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T03:49:36.667 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T03:49:36.669 INFO:tasks.ceph.osd.3:Started 2024-09-16T03:49:36.670 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T03:49:36.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T03:49:36.672 INFO:tasks.ceph.osd.4:Started 2024-09-16T03:49:36.672 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T03:49:36.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T03:49:36.674 INFO:tasks.ceph.osd.5:Started 2024-09-16T03:49:36.674 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T03:49:36.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T03:49:36.676 INFO:tasks.ceph.osd.6:Started 2024-09-16T03:49:36.676 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T03:49:36.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T03:49:36.679 INFO:tasks.ceph.osd.7:Started 2024-09-16T03:49:36.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:36.692 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:36.689+0000 7f37f6412040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T03:49:36.765 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:36.762+0000 7f37f6412040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T03:49:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.805+0000 7f270825a640 10 monclient: get_monmap_and_config 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.805+0000 7f270825a640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.805+0000 7f270825a640 10 monclient: monmap: 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:36.806679+0000 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:36.806679+0000 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:36.809 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.805+0000 7f270825a640 1 Processor -- start 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 1 -- start start 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient: init 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T03:49:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27000b5650 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27000b3120 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2700146c10 0x7f270014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): picked mon.noname-a con 0x7f2700146c10 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27000aae40 con 0x7f27000b5650 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2700140b20 con 0x7f27000b3120 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f27000b5a30 con 0x7f2700146c10 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f270825a640 10 monclient(hunting): authenticate will time out at 735.468672s 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f2705ff6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f27000b5650 0x7f2700140410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55886/0 (socket says 172.21.15.70:55886) 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f2705ff6640 1 -- 172.21.15.70:0/3604461153 learned_addr learned my addr 172.21.15.70:0/3604461153 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27057f5640 1 --1- 172.21.15.70:0/3604461153 >> v1:172.21.15.163:6789/0 conn(0x7f27000b3120 0x7f2700146500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:59514/0 (socket says 172.21.15.70:59514) 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27067f7640 1 --2- 172.21.15.70:0/3604461153 >> v2:172.21.15.70:3300/0 conn(0x7f2700146c10 0x7f270014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27067f7640 10 monclient(hunting): get_auth_request con 0x7f2700146c10 auth_method 0 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27067f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27067f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.806+0000 7f27067f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient(hunting): handle_auth_done global_id 4188 payload 274 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f2704ff4640 1 -- 172.21.15.70:0/3604461153 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239175480 0 0) 0x7f2700140b20 con 0x7f27000b3120 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 1 -- 172.21.15.70:0/3604461153 >> v1:172.21.15.163:6789/0 conn(0x7f27000b3120 legacy=0x7f2700146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 1 -- 172.21.15.70:0/3604461153 >> v1:172.21.15.70:6789/0 conn(0x7f27000b5650 legacy=0x7f2700140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 1 monclient: found mon.noname-a 2024-09-16T03:49:36.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 1 -- 172.21.15.70:0/3604461153 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27000b5cd0 con 0x7f2700146c10 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f2704ff4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.807+0000 7f27067f7640 1 --2- 172.21.15.70:0/3604461153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2700146c10 0x7f270014e8a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009d90 tx=0x7f26f802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e690ccf33ebf55e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 1 -- 172.21.15.70:0/3604461153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8007740 con 0x7f2700146c10 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: dump: 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.821 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 1 -- 172.21.15.70:0/3604461153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f80078e0 con 0x7f2700146c10 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 1 -- 172.21.15.70:0/3604461153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8007cb0 con 0x7f2700146c10 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:36.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f2704ff4640 10 monclient: dump: 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 5 monclient: authenticate success, global_id 4188 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 4 set_mon_vals no callback set 2024-09-16T03:49:36.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 10 monclient: shutdown 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 -- 172.21.15.70:0/3604461153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2700146c10 msgr2=0x7f270014e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 --2- 172.21.15.70:0/3604461153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2700146c10 0x7f270014e8a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009d90 tx=0x7f26f802a660 comp rx=0 tx=0).stop 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 -- 172.21.15.70:0/3604461153 shutdown_connections 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 --2- 172.21.15.70:0/3604461153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2700146c10 0x7f270014e8a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 -- 172.21.15.70:0/3604461153 >> 172.21.15.70:0/3604461153 conn(0x7f27000679c0 msgr2=0x7f2700067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 -- 172.21.15.70:0/3604461153 shutdown_connections 2024-09-16T03:49:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.808+0000 7f270825a640 1 -- 172.21.15.70:0/3604461153 wait complete. 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.809+0000 7f270825a640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.809+0000 7f270825a640 10 monclient: monmap: 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.809+0000 7f270825a640 1 librados: starting msgr at 2024-09-16T03:49:36.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.809+0000 7f270825a640 1 librados: starting objecter 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 Processor -- start 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 -- start start 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 librados: setting wanted keys 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 librados: calling monclient init 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient: init 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 0x7f27000b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): picked mon.noname-b con 0x7f27000b3120 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27000b5650 0x7f270014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): picked mon.noname-a con 0x7f27000b5650 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): picked mon.noname-c con 0x7f2700146c10 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:36.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27000aab70 con 0x7f27000b5650 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f27000aa9f0 con 0x7f27000b3120 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f270014ede0 con 0x7f2700146c10 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f2705ff6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 0x7f27000b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f27067f7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f270825a640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f27067f7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:35824/0 (socket says 172.21.15.70:35824) 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f2705ff6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 0x7f27000b08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:42560/0 (socket says 172.21.15.70:42560) 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f27067f7640 1 -- 172.21.15.70:0/1637157357 learned_addr learned my addr 172.21.15.70:0/1637157357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.810+0000 7f27057f5640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27000b5650 0x7f270014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 monclient(hunting): authenticate will time out at 735.473022s 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): get_auth_request con 0x7f2700146c10 auth_method 0 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): get_auth_request con 0x7f27000b3120 auth_method 0 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): get_auth_request con 0x7f27000b5650 auth_method 0 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f2705ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27057f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient(hunting): handle_auth_done global_id 4157 payload 824 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 -- 172.21.15.70:0/1637157357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 msgr2=0x7f27000b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 0x7f27000b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 -- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27000b5650 msgr2=0x7f270014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27000b5650 0x7f270014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient: _finish_hunting 0 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 monclient: found mon.noname-c 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 -- 172.21.15.70:0/1637157357 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27000b1590 con 0x7f2700146c10 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f27067f7640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009d90 tx=0x7f26f8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 client.?.objecter ms_handle_connect 0x7f2700146c10 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f803f070 con 0x7f2700146c10 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: dump: 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:36.830 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 5 monclient: authenticate success, global_id 4157 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 monclient: _renew_subs 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f8031bc0 con 0x7f2700146c10 2024-09-16T03:49:36.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27000b0df0 con 0x7f2700146c10 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8043410 con 0x7f2700146c10 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f27fc640 10 monclient: dump: 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:36.832 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f26f3fff640 4 set_mon_vals no callback set 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 monclient: _renew_subs 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2700147650 con 0x7f2700146c10 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.811+0000 7f270825a640 1 librados: init done 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f26f8043940 con 0x7f2700146c10 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f26f8007840 con 0x7f2700146c10 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 10 client.4157.objecter ms_dispatch 0x7f2700000b90 osd_map(10..10 src has 1..10) 2024-09-16T03:49:36.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f26f27fc640 20 client.4157.objecter dump_active .. 0 homeless 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.812+0000 7f270825a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.813+0000 7f270825a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.813+0000 7f270825a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.813+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2700147880 con 0x7f2700146c10 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.815+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f26f8043c30 con 0x7f2700146c10 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.815+0000 7f26f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:36.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.815+0000 7f26f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:36.850 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:36.848+0000 7f14277b9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.873+0000 7f270825a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.873+0000 7f270825a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.873+0000 7f270825a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.873+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2700146500 con 0x7f2700146c10 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.874+0000 7f26f27fc640 1 -- 172.21.15.70:0/1637157357 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f26f803b410 con 0x7f2700146c10 2024-09-16T03:49:36.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.874+0000 7f26f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:36.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.874+0000 7f26f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:36.877 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:49:36.877 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":10,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:36.644668+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:36.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 10 librados: watch_flush enter 2024-09-16T03:49:36.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 10 librados: watch_flush exit 2024-09-16T03:49:36.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 20 client.4157.objecter shutdown clearing up homeless session... 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 10 client.4157.objecter successfully canceled tick 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 10 monclient: shutdown 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 msgr2=0x7f270014e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009d90 tx=0x7f26f8004a40 comp rx=0 tx=0).stop 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 shutdown_connections 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2700146c10 0x7f270014e770 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27000b5650 0x7f270014e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 --2- 172.21.15.70:0/1637157357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f27000b3120 0x7f27000b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 >> 172.21.15.70:0/1637157357 conn(0x7f27000679c0 msgr2=0x7f270013e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 shutdown_connections 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 -- 172.21.15.70:0/1637157357 wait complete. 2024-09-16T03:49:36.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:36.875+0000 7f270825a640 1 librados: shutdown 2024-09-16T03:49:36.888 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T03:49:36.889 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T03:49:36.902 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:36.900+0000 7f14277b9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T03:49:37.138 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.136+0000 7f37f6412040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T03:49:37.222 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.219+0000 7f37f6412040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T03:49:37.271 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.270+0000 7f14277b9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T03:49:37.330 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.328+0000 7f37f6412040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:49:37.359 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.357+0000 7f14277b9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T03:49:37.377 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.375+0000 7f37f6412040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T03:49:37.465 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.464+0000 7f14277b9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:49:37.489 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.486+0000 7f37f6412040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T03:49:37.511 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.509+0000 7f14277b9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T03:49:37.541 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.538+0000 7f37f6412040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T03:49:37.588 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.585+0000 7f37f6412040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T03:49:37.627 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.625+0000 7f14277b9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T03:49:37.680 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.678+0000 7f14277b9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T03:49:37.693 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:37.700 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.698+0000 7f37f6412040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T03:49:37.724 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.722+0000 7f14277b9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T03:49:37.753 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.750+0000 7f37f6412040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T03:49:37.783 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.781+0000 7fa55fea6640 10 monclient: get_monmap_and_config 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.781+0000 7fa55fea6640 10 monclient: build_initial_monmap 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.781+0000 7fa55fea6640 10 monclient: monmap: 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:49:37.782794+0000 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:49:37.782794+0000 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:37.784 INFO:teuthology.misc.health.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:37.785 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.781+0000 7fa55fea6640 1 Processor -- start 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 -- start start 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient: init 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 0x7fa558146560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5580b5db0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:37.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 0x7fa5580aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5580b3210 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5581401e0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5580ac2e0 con 0x7fa5581401e0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fa5580b6190 con 0x7fa5580b3210 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fa5580aa440 con 0x7fa5580b5db0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55fea6640 10 monclient(hunting): authenticate will time out at 736.444775s 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 0x7fa558146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55e443640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa5581401e0 0x7fa558140690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55892/0 (socket says 172.21.15.70:55892) 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55e443640 1 -- 172.21.15.70:0/678376925 learned_addr learned my addr 172.21.15.70:0/678376925 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 10 monclient(hunting): get_auth_request con 0x7fa5580b5db0 auth_method 0 2024-09-16T03:49:37.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.782+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 1 --2- 172.21.15.70:0/678376925 >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 0x7fa5580aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 10 monclient(hunting): get_auth_request con 0x7fa5580b3210 auth_method 0 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 -- 172.21.15.70:0/678376925 >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 msgr2=0x7fa5580aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 --2- 172.21.15.70:0/678376925 >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 0x7fa5580aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 -- 172.21.15.70:0/678376925 >> v1:172.21.15.70:6789/0 conn(0x7fa5581401e0 legacy=0x7fa558140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 10 monclient: _finish_hunting 0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 monclient: found mon.noname-b 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 -- 172.21.15.70:0/678376925 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5580aa720 con 0x7fa5580b5db0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55cc40640 1 -- 172.21.15.70:0/678376925 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254528230 0 0) 0x7fa5580ac2e0 con 0x7fa5581401e0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55d441640 1 --2- 172.21.15.70:0/678376925 >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 0x7fa5580aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55cc40640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.783+0000 7fa55dc42640 1 --2- 172.21.15.70:0/678376925 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 0x7fa558146560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa5540097a0 tx=0x7fa55402a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce90ba5f4537e0d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 1 -- 172.21.15.70:0/678376925 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554037070 con 0x7fa5580b5db0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:37.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: dump: 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: _finish_auth 0 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: _check_auth_tickets 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:37.796 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 5 monclient: authenticate success, global_id 4160 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 1 -- 172.21.15.70:0/678376925 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55402adb0 con 0x7fa5580b5db0 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 1 -- 172.21.15.70:0/678376925 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554032660 con 0x7fa5580b5db0 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 4 set_mon_vals no callback set 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55cc40640 10 monclient: dump: 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.797 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 10 monclient: shutdown 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 -- 172.21.15.70:0/678376925 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 msgr2=0x7fa558146560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 --2- 172.21.15.70:0/678376925 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 0x7fa558146560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa5540097a0 tx=0x7fa55402a060 comp rx=0 tx=0).stop 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 -- 172.21.15.70:0/678376925 shutdown_connections 2024-09-16T03:49:37.798 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 --2- 172.21.15.70:0/678376925 >> v2:172.21.15.163:3300/0 conn(0x7fa5580b3210 0x7fa5580aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 --2- 172.21.15.70:0/678376925 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b5db0 0x7fa558146560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 -- 172.21.15.70:0/678376925 >> 172.21.15.70:0/678376925 conn(0x7fa55806cc00 msgr2=0x7fa55806d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 -- 172.21.15.70:0/678376925 shutdown_connections 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.784+0000 7fa55fea6640 1 -- 172.21.15.70:0/678376925 wait complete. 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 10 monclient: build_initial_monmap 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 10 monclient: monmap: 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:created 0.000000 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:37.799 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 librados: starting msgr at 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 librados: starting objecter 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 Processor -- start 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 -- start start 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 librados: setting wanted keys 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 1 librados: calling monclient init 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.785+0000 7fa55fea6640 10 monclient: init 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5580b3210 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:37.800 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5580b5db0 0x7fa558147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5580b5db0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5581401e0 0x7fa558147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): picked mon.noname-a con 0x7fa5581401e0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5580b56c0 con 0x7fa5581401e0 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa5580b5540 con 0x7fa5580b5db0 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fa55814d8b0 con 0x7fa5580b3210 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55fea6640 10 monclient(hunting): authenticate will time out at 736.448359s 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:35844/0 (socket says 172.21.15.70:35844) 2024-09-16T03:49:37.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 -- 172.21.15.70:0/871505529 learned_addr learned my addr 172.21.15.70:0/871505529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): get_auth_request con 0x7fa5580b3210 auth_method 0 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5581401e0 0x7fa558147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): get_auth_request con 0x7fa5581401e0 auth_method 0 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:37.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5580b5db0 0x7fa558147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): get_auth_request con 0x7fa5580b5db0 auth_method 0 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55e443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55d441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-09-16T03:49:37.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 -- 172.21.15.70:0/871505529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5580b5db0 msgr2=0x7fa558147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5580b5db0 0x7fa558147670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 -- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5581401e0 msgr2=0x7fa558147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5581401e0 0x7fa558147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient: _finish_hunting 0 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 monclient: found mon.noname-c 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 -- 172.21.15.70:0/871505529 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55814c190 con 0x7fa5580b3210 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa55dc42640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098c0 tx=0x7fa5540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa54e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5580b3210 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.786+0000 7fa54e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554041030 con 0x7fa5580b3210 2024-09-16T03:49:37.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: dump: 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:37.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 5 monclient: authenticate success, global_id 4163 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55403c670 con 0x7fa5580b3210 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554037040 con 0x7fa5580b3210 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54ffff640 4 set_mon_vals no callback set 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 monclient: dump: 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:37.806 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 10 monclient: _renew_subs 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55814c470 con 0x7fa5580b3210 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 10 monclient: _renew_subs 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55814a7d0 con 0x7fa5580b3210 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa55fea6640 1 librados: init done 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa554044070 con 0x7fa5580b3210 2024-09-16T03:49:37.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa554046e00 con 0x7fa5580b3210 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 10 client.4163.objecter ms_dispatch 0x7fa558000b90 osd_map(10..10 src has 1..10) 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 3 client.4163.objecter handle_osd_map got epochs [10,10] > 0 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 3 client.4163.objecter handle_osd_map decoding full epoch 10 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.787+0000 7fa54e7fc640 20 client.4163.objecter dump_active .. 0 homeless 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.788+0000 7fa55fea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.788+0000 7fa55fea6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.788+0000 7fa55fea6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.788+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5580aac40 con 0x7fa5580b3210 2024-09-16T03:49:37.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.791+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fa55402ac70 con 0x7fa5580b3210 2024-09-16T03:49:37.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.791+0000 7fa54e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:37.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.791+0000 7fa54e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:37.809 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.806+0000 7f37f6412040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T03:49:37.837 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.834+0000 7f14277b9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T03:49:37.853 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.851+0000 7fa55fea6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:37.853 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.851+0000 7fa55fea6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:37.854 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.851+0000 7fa55fea6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:37.854 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.851+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa55814d690 con 0x7fa5580b3210 2024-09-16T03:49:37.854 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa54e7fc640 1 -- 172.21.15.70:0/871505529 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa55403ca40 con 0x7fa5580b3210 2024-09-16T03:49:37.854 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa54e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:37.854 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa54e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:37.855 INFO:teuthology.misc.health.smithi070.stdout: 2024-09-16T03:49:37.855 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":10,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:36.644668+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:37.856 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 10 librados: watch_flush enter 2024-09-16T03:49:37.856 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 10 librados: watch_flush exit 2024-09-16T03:49:37.856 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 20 client.4163.objecter shutdown clearing up homeless session... 2024-09-16T03:49:37.856 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 10 client.4163.objecter successfully canceled tick 2024-09-16T03:49:37.856 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 10 monclient: shutdown 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 msgr2=0x7fa55814a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098c0 tx=0x7fa5540046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 shutdown_connections 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5581401e0 0x7fa558147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa5580b5db0 0x7fa558147670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 --2- 172.21.15.70:0/871505529 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa5580b3210 0x7fa55814a010 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.852+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 >> 172.21.15.70:0/871505529 conn(0x7fa55806cc00 msgr2=0x7fa558140690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.853+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 shutdown_connections 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.853+0000 7fa55fea6640 1 -- 172.21.15.70:0/871505529 wait complete. 2024-09-16T03:49:37.857 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:37.853+0000 7fa55fea6640 1 librados: shutdown 2024-09-16T03:49:37.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T03:49:37.879 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.877+0000 7f37f6412040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T03:49:37.888 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.887+0000 7f14277b9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T03:49:37.892 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-09-16T03:49:37.889+0000 7f8494e16640 -1 Falling back to public interface 2024-09-16T03:49:37.925 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.922+0000 7f37f6412040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T03:49:37.941 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:37.939+0000 7f14277b9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T03:49:37.969 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:37.966+0000 7f37f6412040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T03:49:38.007 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:38.005+0000 7f14277b9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T03:49:38.051 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:38.049+0000 7f14277b9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T03:49:38.094 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:38.093+0000 7f14277b9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T03:49:38.145 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T03:49:38.143+0000 7f9328b51640 -1 Falling back to public interface 2024-09-16T03:49:38.395 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T03:49:38.393+0000 7f832d00b640 -1 Falling back to public interface 2024-09-16T03:49:38.625 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:38.623+0000 7f37f6412040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T03:49:38.640 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:49:38.637+0000 7fe03fa0d640 -1 Falling back to public interface 2024-09-16T03:49:38.649 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-16T03:49:38.646+0000 7f4160c15640 -1 Falling back to public interface 2024-09-16T03:49:38.650 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-09-16T03:49:38.647+0000 7f9cf980c640 -1 Falling back to public interface 2024-09-16T03:49:38.651 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T03:49:38.650+0000 7f9327e14640 -1 Falling back to public interface 2024-09-16T03:49:38.661 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T03:49:38.659+0000 7fea4b014640 -1 Falling back to public interface 2024-09-16T03:49:38.765 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-09-16T03:49:38.764+0000 7f14277b9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T03:49:39.641 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:39.640+0000 7f3796016640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T03:49:39.642 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-09-16T03:49:39.640+0000 7f3796016640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T03:49:42.460 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-09-16T03:49:42.458+0000 7f8494e16640 -1 osd.0 0 log_to_monitors true 2024-09-16T03:49:42.706 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:49:42.704+0000 7fe03fa0d640 -1 osd.3 0 log_to_monitors true 2024-09-16T03:49:42.711 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T03:49:42.709+0000 7f9328b51640 -1 osd.7 0 log_to_monitors true 2024-09-16T03:49:42.951 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T03:49:42.950+0000 7f832d00b640 -1 osd.4 0 log_to_monitors true 2024-09-16T03:49:42.963 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T03:49:42.962+0000 7fea4b014640 -1 osd.6 0 log_to_monitors true 2024-09-16T03:49:42.981 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-09-16T03:49:42.979+0000 7f9cf980c640 -1 osd.1 0 log_to_monitors true 2024-09-16T03:49:43.130 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T03:49:43.128+0000 7f9327e14640 -1 osd.5 0 log_to_monitors true 2024-09-16T03:49:43.238 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-16T03:49:43.237+0000 7f4160c15640 -1 osd.2 0 log_to_monitors true 2024-09-16T03:49:44.257 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-09-16T03:49:44.256+0000 7f8490fae640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T03:49:44.260 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-09-16T03:49:44.258+0000 7f84885a9640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:44.660 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-09-16T03:49:44.658+0000 7f9cf49a5640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T03:49:44.660 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T03:49:44.659+0000 7f93242bd640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T03:49:44.663 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T03:49:44.662+0000 7f9322e7c640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T03:49:44.670 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:44.678 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-09-16T03:49:44.677+0000 7f9cf0fdf640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:44.783 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.781+0000 7f268a113640 10 monclient: get_monmap_and_config 2024-09-16T03:49:44.783 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.781+0000 7f268a113640 10 monclient: build_initial_monmap 2024-09-16T03:49:44.783 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.781+0000 7f268a113640 10 monclient: monmap: 2024-09-16T03:49:44.783 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:44.783 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:49:44.782295+0000 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:49:44.782295+0000 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:44.784 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.782+0000 7f268a113640 1 Processor -- start 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.782+0000 7f268a113640 1 -- start start 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.782+0000 7f268a113640 10 monclient: init 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26840b5db0 0x7f2684146560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): picked mon.noname-a con 0x7f26840b5db0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 0x7f26840aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.788 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): picked mon.noname-c con 0x7f26840b3210 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26841401e0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26840ac2e0 con 0x7f26841401e0 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f26840b6190 con 0x7f26840b5db0 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f26840aa440 con 0x7f26840b3210 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f268a113640 10 monclient(hunting): authenticate will time out at 743.444676s 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26840b5db0 0x7f2684146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26840b5db0 0x7f2684146560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41320/0 (socket says 172.21.15.70:41320) 2024-09-16T03:49:44.789 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 1 -- 172.21.15.70:0/2400183092 learned_addr learned my addr 172.21.15.70:0/2400183092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): get_auth_request con 0x7f26840b5db0 auth_method 0 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f2682ffd640 1 --2- 172.21.15.70:0/2400183092 >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 0x7f26840aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f2682ffd640 10 monclient(hunting): get_auth_request con 0x7f26840b3210 auth_method 0 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f2682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f2682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f2682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.783+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:44.790 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 -- 172.21.15.70:0/2400183092 >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 msgr2=0x7f26840aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 --2- 172.21.15.70:0/2400183092 >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 0x7f26840aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 -- 172.21.15.70:0/2400183092 >> v1:172.21.15.163:6789/0 conn(0x7f26841401e0 legacy=0x7f2684140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 -- 172.21.15.70:0/2400183092 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26840aa720 con 0x7f26840b5db0 2024-09-16T03:49:44.791 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f2682ffd640 1 --2- 172.21.15.70:0/2400183092 >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 0x7f26840aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.784+0000 7f26837fe640 1 --2- 172.21.15.70:0/2400183092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26840b5db0 0x7f2684146560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f26840b9dd0 tx=0x7f267402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5696da05b92d7727 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 1 -- 172.21.15.70:0/2400183092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674037070 con 0x7f26840b5db0 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: dump: 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:44.792 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 1 -- 172.21.15.70:0/2400183092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2674032440 con 0x7f26840b5db0 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:44.793 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 1 -- 172.21.15.70:0/2400183092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674032810 con 0x7f26840b5db0 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f26827fc640 10 monclient: dump: 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:44.794 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f268a113640 5 monclient: authenticate success, global_id 4242 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f268a113640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.785+0000 7f268a113640 4 set_mon_vals no callback set 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 10 monclient: shutdown 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 -- 172.21.15.70:0/2400183092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26840b5db0 msgr2=0x7f2684146560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 --2- 172.21.15.70:0/2400183092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26840b5db0 0x7f2684146560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f26840b9dd0 tx=0x7f267402a150 comp rx=0 tx=0).stop 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 -- 172.21.15.70:0/2400183092 shutdown_connections 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 --2- 172.21.15.70:0/2400183092 >> v2:172.21.15.163:3300/0 conn(0x7f26840b3210 0x7f26840aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 --2- 172.21.15.70:0/2400183092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26840b5db0 0x7f2684146560 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 -- 172.21.15.70:0/2400183092 >> 172.21.15.70:0/2400183092 conn(0x7f268406cc00 msgr2=0x7f268406d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 -- 172.21.15.70:0/2400183092 shutdown_connections 2024-09-16T03:49:44.795 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.786+0000 7f268a113640 1 -- 172.21.15.70:0/2400183092 wait complete. 2024-09-16T03:49:44.796 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-09-16T03:49:44.793+0000 7f93208f7640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:44.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 10 monclient: build_initial_monmap 2024-09-16T03:49:44.801 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 10 monclient: monmap: 2024-09-16T03:49:44.801 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:44.801 INFO:teuthology.misc.health.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:44.801 INFO:teuthology.misc.health.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:created 0.000000 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 librados: starting msgr at 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 librados: starting objecter 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 Processor -- start 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 -- start start 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 librados: setting wanted keys 2024-09-16T03:49:44.802 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 1 librados: calling monclient init 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.787+0000 7f268a113640 10 monclient: init 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 0x7f268413f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): picked mon.noname-c con 0x7f26840b3210 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): picked mon.noname-b con 0x7f26840b5db0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 0x7f2684149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): picked mon.noname-a con 0x7f26841401e0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:44.803 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26840b56c0 con 0x7f26841401e0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26840b5540 con 0x7f26840b5db0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f268414d8b0 con 0x7f26840b3210 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f268a113640 10 monclient(hunting): authenticate will time out at 743.449594s 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 0x7f268413f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 0x7f2684149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:56362/0 (socket says 172.21.15.70:56362) 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 0x7f268413f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46486/0 (socket says 172.21.15.70:46486) 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 1 -- 172.21.15.70:0/1309829061 learned_addr learned my addr 172.21.15.70:0/1309829061 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): get_auth_request con 0x7f26840b5db0 auth_method 0 2024-09-16T03:49:44.804 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): get_auth_request con 0x7f26841401e0 auth_method 0 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): get_auth_request con 0x7f26840b3210 auth_method 0 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:44.805 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f2683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.788+0000 7f26837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 -- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 msgr2=0x7f268413f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 0x7f268413f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 -- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 msgr2=0x7f2684149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 0x7f2684149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:44.806 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 monclient: found mon.noname-b 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 -- 172.21.15.70:0/1309829061 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268414b3c0 con 0x7f26840b5db0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2683fff640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 0x7f2684149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2682ffd640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f267800e9e0 tx=0x7f267800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 client.?.objecter ms_handle_connect 0x7f26840b5db0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678017070 con 0x7f26840b5db0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: dump: 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:44.807 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: _finish_auth 0 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 5 monclient: authenticate success, global_id 4237 2024-09-16T03:49:44.808 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 10 monclient: _renew_subs 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267800ca30 con 0x7f26840b5db0 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268414c700 con 0x7f26840b5db0 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267800ce20 con 0x7f26840b5db0 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f2681ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f266bfff640 10 monclient: dump: 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:44.809 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 10 monclient: _renew_subs 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268414bd30 con 0x7f26840b5db0 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.789+0000 7f268a113640 1 librados: init done 2024-09-16T03:49:44.810 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.791+0000 7f268a113640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.791+0000 7f268a113640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.791+0000 7f268a113640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.791+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26841492b0 con 0x7f26840b5db0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.791+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 4) ==== 90755+0+0 (secure 0 0 0) 0x7f267801a070 con 0x7f26840b5db0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 1 --2- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 0x7f2670070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f26837fe640 1 --2- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 0x7f2670070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f26837fe640 10 monclient: get_auth_request con 0x7f267006db70 auth_method 0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2371+0+0 (secure 0 0 0) 0x7f267808a200 con 0x7f26840b5db0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 10 client.4237.objecter ms_dispatch 0x7f2684000b90 osd_map(13..13 src has 1..13) 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 3 client.4237.objecter handle_osd_map got epochs [13,13] > 0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 3 client.4237.objecter handle_osd_map decoding full epoch 13 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 20 client.4237.objecter dump_active .. 0 homeless 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f266bfff640 10 client.4237.objecter ms_handle_connect 0x7f267006db70 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.792+0000 7f26837fe640 1 --2- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 0x7f2670070030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2674034000 tx=0x7f267402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.797+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2678059060 con 0x7f26840b5db0 2024-09-16T03:49:44.811 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.797+0000 7f266bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:44.812 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:44.797+0000 7f266bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:44.812 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-09-16T03:49:44.806+0000 7f931f4b6640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:45.028 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f268a113640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:45.028 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f268a113640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:45.028 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f268a113640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26840b2aa0 con 0x7f26840b5db0 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f266bfff640 1 -- 172.21.15.70:0/1309829061 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7315 (secure 0 0 0) 0x7f267805d360 con 0x7f26840b5db0 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f266bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.027+0000 7f266bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stdout: 2024-09-16T03:49:45.029 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":13,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:44.649819+0000","last_up_change":"2024-09-16T03:49:44.649819+0000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1243681362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1243681362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":1243681362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1243681362}]},"public_addr":"172.21.15.70:6800/1243681362","cluster_addr":"172.21.15.70:6801/1243681362","heartbeat_back_addr":"172.21.15.70:6803/1243681362","heartbeat_front_addr":"172.21.15.70:6802/1243681362","state":["exists","up"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:45.032 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 10 librados: watch_flush enter 2024-09-16T03:49:45.032 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 10 librados: watch_flush exit 2024-09-16T03:49:45.032 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 20 client.4237.objecter shutdown clearing up homeless session... 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 10 client.4237.objecter successfully canceled tick 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 msgr2=0x7f2670070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 0x7f2670070030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2674034000 tx=0x7f267402e000 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 10 monclient: shutdown 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 msgr2=0x7f2684149dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f267800e9e0 tx=0x7f267800eea0 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 shutdown_connections 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> v2:172.21.15.70:6804/198622594 conn(0x7f267006db70 0x7f2670070030 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26841401e0 0x7f2684149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26840b5db0 0x7f2684149dc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 --2- 172.21.15.70:0/1309829061 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f26840b3210 0x7f268413f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:45.033 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.030+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 >> 172.21.15.70:0/1309829061 conn(0x7f268406cc00 msgr2=0x7f268414cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:45.034 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.031+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 shutdown_connections 2024-09-16T03:49:45.034 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.031+0000 7f268a113640 1 -- 172.21.15.70:0/1309829061 wait complete. 2024-09-16T03:49:45.034 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:45.031+0000 7f268a113640 1 librados: shutdown 2024-09-16T03:49:45.043 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-09-16T03:49:45.456 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:49:45.455+0000 7fe03aafb640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T03:49:45.458 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-16T03:49:45.456+0000 7f415bcd2640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T03:49:45.459 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:49:45.458+0000 7fe03312d640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:45.460 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-16T03:49:45.459+0000 7f4154304640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:45.818 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T03:49:45.817+0000 7f8328228640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T03:49:45.821 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T03:49:45.820+0000 7f832085a640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:45.822 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T03:49:45.820+0000 7fea46177640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T03:49:45.826 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-09-16T03:49:45.824+0000 7fea3e7a9640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:49:51.850 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:51.950 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.949+0000 7f22de931640 10 monclient: get_monmap_and_config 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.949+0000 7f22de931640 10 monclient: build_initial_monmap 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.949+0000 7f22de931640 10 monclient: monmap: 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:49:51.949686+0000 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:49:51.949686+0000 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:51.951 INFO:teuthology.misc.health.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:51.952 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:51.952 INFO:teuthology.misc.health.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:51.952 INFO:teuthology.misc.health.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:51.952 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.960 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 Processor -- start 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 -- start start 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient: init 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22d80b55d0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22d81408d0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 0x7f22d80b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): picked mon.noname-b con 0x7f22d80b3090 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d80ab1e0 con 0x7f22d81408d0 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d80aacc0 con 0x7f22d80b55d0 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:51.961 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f22d80aa900 con 0x7f22d80b3090 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22de931640 10 monclient(hunting): authenticate will time out at 750.611777s 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22d77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f22d81408d0 0x7f22d8146460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59282/0 (socket says 172.21.15.70:59282) 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22dcece640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 0x7f22d80b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22d77fe640 1 -- 172.21.15.70:0/3587989980 learned_addr learned my addr 172.21.15.70:0/3587989980 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22dcece640 10 monclient(hunting): get_auth_request con 0x7f22d80b3090 auth_method 0 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22dcece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22dcece640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.950+0000 7f22dcece640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:51.962 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 1 -- 172.21.15.70:0/3587989980 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817459584 0 0) 0x7f22d80ab1e0 con 0x7f22d81408d0 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 10 monclient(hunting): my global_id is 4257 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22d6ffd640 1 -- 172.21.15.70:0/3587989980 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c4003520 con 0x7f22d81408d0 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 1 -- 172.21.15.70:0/3587989980 >> v1:172.21.15.163:6789/0 conn(0x7f22d80b55d0 legacy=0x7f22d81401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 1 -- 172.21.15.70:0/3587989980 >> v1:172.21.15.70:6789/0 conn(0x7f22d81408d0 legacy=0x7f22d8146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient: _finish_hunting 0 2024-09-16T03:49:51.963 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 1 monclient: found mon.noname-b 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 1 -- 172.21.15.70:0/3587989980 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d8146b70 con 0x7f22d80b3090 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.951+0000 7f22dcece640 1 --2- 172.21.15.70:0/3587989980 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 0x7f22d80b59b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009dc0 tx=0x7f22d002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78daab23d4caa89e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 1 -- 172.21.15.70:0/3587989980 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0007740 con 0x7f22d80b3090 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: dump: 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.964 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 1 -- 172.21.15.70:0/3587989980 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d00078e0 con 0x7f22d80b3090 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 5 monclient: authenticate success, global_id 4220 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:51.965 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:51.966 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 1 -- 172.21.15.70:0/3587989980 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0007cb0 con 0x7f22d80b3090 2024-09-16T03:49:51.966 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 4 set_mon_vals no callback set 2024-09-16T03:49:51.966 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:51.966 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:51.969 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22d6ffd640 10 monclient: dump: 2024-09-16T03:49:51.969 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:51.969 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:51.970 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 10 monclient: shutdown 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 -- 172.21.15.70:0/3587989980 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 msgr2=0x7f22d80b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 --2- 172.21.15.70:0/3587989980 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 0x7f22d80b59b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22d0009dc0 tx=0x7f22d002a6d0 comp rx=0 tx=0).stop 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 -- 172.21.15.70:0/3587989980 shutdown_connections 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 --2- 172.21.15.70:0/3587989980 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d80b3090 0x7f22d80b59b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 -- 172.21.15.70:0/3587989980 >> 172.21.15.70:0/3587989980 conn(0x7f22d8067aa0 msgr2=0x7f22d8067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 -- 172.21.15.70:0/3587989980 shutdown_connections 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.952+0000 7f22de931640 1 -- 172.21.15.70:0/3587989980 wait complete. 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.953+0000 7f22de931640 10 monclient: build_initial_monmap 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.953+0000 7f22de931640 10 monclient: monmap: 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:epoch 0 2024-09-16T03:49:51.971 INFO:teuthology.misc.health.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:created 0.000000 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.953+0000 7f22de931640 1 librados: starting msgr at 2024-09-16T03:49:51.972 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.953+0000 7f22de931640 1 librados: starting objecter 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.953+0000 7f22de931640 1 Processor -- start 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 -- start start 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 librados: setting wanted keys 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 librados: calling monclient init 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient: init 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 0x7f22d81432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): picked mon.noname-b con 0x7f22d80b3090 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 0x7f22d80aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): picked mon.noname-a con 0x7f22d80b55d0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:51.973 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): picked mon.noname-c con 0x7f22d81408d0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f22d814e800 con 0x7f22d80b55d0 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22d80aaa80 con 0x7f22d80b3090 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f22d814e980 con 0x7f22d81408d0 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22de931640 10 monclient(hunting): authenticate will time out at 750.615593s 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 0x7f22d80aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46520/0 (socket says 172.21.15.70:46520) 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 1 -- 172.21.15.70:0/1296295989 learned_addr learned my addr 172.21.15.70:0/1296295989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): get_auth_request con 0x7f22d81408d0 auth_method 0 2024-09-16T03:49:51.974 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): get_auth_request con 0x7f22d80b55d0 auth_method 0 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:51.975 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22d7fff640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 0x7f22d81432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.954+0000 7f22dcece640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 -- 172.21.15.70:0/1296295989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 msgr2=0x7f22d81432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 0x7f22d81432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 -- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 msgr2=0x7f22d80aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 0x7f22d80aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 10 monclient: _finish_hunting 0 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 monclient: found mon.noname-c 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:51.976 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 -- 172.21.15.70:0/1296295989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d80af6d0 con 0x7f22d81408d0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d7fff640 10 monclient: get_auth_request con 0x7f22d80b3090 auth_method 0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d7fff640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 0x7f22d81432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d77fe640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 0x7f22d80aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22dcece640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f22d002acc0 tx=0x7f22d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 client.?.objecter ms_handle_connect 0x7f22d81408d0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0040070 con 0x7f22d81408d0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: dump: 2024-09-16T03:49:51.977 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:51.978 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 5 monclient: authenticate success, global_id 4223 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d0031bc0 con 0x7f22d81408d0 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 10 monclient: _renew_subs 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d80b1360 con 0x7f22d81408d0 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0045460 con 0x7f22d81408d0 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d4ff9640 10 monclient: dump: 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:epoch 1 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.979 INFO:teuthology.misc.health.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr: 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22d67fc640 4 set_mon_vals no callback set 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 10 monclient: _renew_subs 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d80afaf0 con 0x7f22d81408d0 2024-09-16T03:49:51.980 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.955+0000 7f22de931640 1 librados: init done 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.956+0000 7f22de931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.956+0000 7f22de931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.956+0000 7f22de931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.956+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d80b1750 con 0x7f22d81408d0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.956+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f22d0039040 con 0x7f22d81408d0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 1 --2- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 0x7f22b80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3737+0+0 (secure 0 0 0) 0x7f22d00ac400 con 0x7f22d81408d0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 10 client.4223.objecter ms_dispatch 0x7f22d8000b90 osd_map(17..17 src has 1..17) 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 3 client.4223.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 3 client.4223.objecter handle_osd_map decoding full epoch 17 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d7fff640 1 --2- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 0x7f22b80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d7fff640 10 monclient: get_auth_request con 0x7f22b806dc00 auth_method 0 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 20 client.4223.objecter dump_active .. 0 homeless 2024-09-16T03:49:51.981 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d7fff640 1 --2- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 0x7f22b80700c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f22c8001680 tx=0x7f22c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:51.982 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.957+0000 7f22d4ff9640 10 client.4223.objecter ms_handle_connect 0x7f22b806dc00 2024-09-16T03:49:51.982 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.961+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f22d007b3b0 con 0x7f22d81408d0 2024-09-16T03:49:51.982 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.961+0000 7f22d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:51.982 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:51.961+0000 7f22d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:52.135 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22de931640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22de931640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22de931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22d80b59b0 con 0x7f22d81408d0 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22d4ff9640 1 -- 172.21.15.70:0/1296295989 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+10984 (secure 0 0 0) 0x7f22d007acb0 con 0x7f22d81408d0 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:52.136 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.134+0000 7f22d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:52.137 INFO:teuthology.misc.health.smithi070.stdout: 2024-09-16T03:49:52.137 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":17,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:48.826352+0000","last_up_change":"2024-09-16T03:49:46.819202+0000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:49:46.663530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1243681362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1243681362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":1243681362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1243681362}]},"public_addr":"172.21.15.70:6800/1243681362","cluster_addr":"172.21.15.70:6801/1243681362","heartbeat_back_addr":"172.21.15.70:6803/1243681362","heartbeat_front_addr":"172.21.15.70:6802/1243681362","state":["exists","up"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":1499120614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":1499120614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1499120614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":1499120614}]},"public_addr":"172.21.15.70:6813/1499120614","cluster_addr":"172.21.15.70:6814/1499120614","heartbeat_back_addr":"172.21.15.70:6816/1499120614","heartbeat_front_addr":"172.21.15.70:6815/1499120614","state":["exists","up"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":2584301924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2584301924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2584301924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":2584301924}]},"public_addr":"172.21.15.70:6809/2584301924","cluster_addr":"172.21.15.70:6810/2584301924","heartbeat_back_addr":"172.21.15.70:6812/2584301924","heartbeat_front_addr":"172.21.15.70:6811/2584301924","state":["exists","up"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3986044637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3986044637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3986044637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3986044637}]},"public_addr":"172.21.15.70:6805/3986044637","cluster_addr":"172.21.15.70:6806/3986044637","heartbeat_back_addr":"172.21.15.70:6808/3986044637","heartbeat_front_addr":"172.21.15.70:6807/3986044637","state":["exists","up"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":1904387775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":1904387775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":1904387775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":1904387775}]},"public_addr":"172.21.15.163:6804/1904387775","cluster_addr":"172.21.15.163:6805/1904387775","heartbeat_back_addr":"172.21.15.163:6807/1904387775","heartbeat_front_addr":"172.21.15.163:6806/1904387775","state":["exists","up"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1625318059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":1625318059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":1625318059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1625318059}]},"public_addr":"172.21.15.163:6808/1625318059","cluster_addr":"172.21.15.163:6809/1625318059","heartbeat_back_addr":"172.21.15.163:6811/1625318059","heartbeat_front_addr":"172.21.15.163:6810/1625318059","state":["exists","up"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":2999981402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":2999981402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":2999981402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":2999981402}]},"public_addr":"172.21.15.163:6812/2999981402","cluster_addr":"172.21.15.163:6813/2999981402","heartbeat_back_addr":"172.21.15.163:6815/2999981402","heartbeat_front_addr":"172.21.15.163:6814/2999981402","state":["exists","up"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3710554613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3710554613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3710554613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3710554613}]},"public_addr":"172.21.15.163:6800/3710554613","cluster_addr":"172.21.15.163:6801/3710554613","heartbeat_back_addr":"172.21.15.163:6803/3710554613","heartbeat_front_addr":"172.21.15.163:6802/3710554613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.496124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.974891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.248106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.668425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.919888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.102132+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.952245+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.672498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.136+0000 7f22de931640 10 librados: watch_flush enter 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 10 librados: watch_flush exit 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 20 client.4223.objecter shutdown clearing up homeless session... 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 10 client.4223.objecter successfully canceled tick 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 msgr2=0x7f22b80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 0x7f22b80700c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f22c8001680 tx=0x7f22c8009040 comp rx=0 tx=0).stop 2024-09-16T03:49:52.139 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 10 monclient: shutdown 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 msgr2=0x7f22d80b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f22d002acc0 tx=0x7f22d0004a40 comp rx=0 tx=0).stop 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 shutdown_connections 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> v2:172.21.15.70:6804/198622594 conn(0x7f22b806dc00 0x7f22b80700c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f22d81408d0 0x7f22d80b1e40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22d80b55d0 0x7f22d80aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 --2- 172.21.15.70:0/1296295989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22d80b3090 0x7f22d81432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 >> 172.21.15.70:0/1296295989 conn(0x7f22d8067aa0 msgr2=0x7f22d80b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 shutdown_connections 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 -- 172.21.15.70:0/1296295989 wait complete. 2024-09-16T03:49:52.140 INFO:teuthology.misc.health.smithi070.stderr:2024-09-16T03:49:52.137+0000 7f22de931640 1 librados: shutdown 2024-09-16T03:49:52.148 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T03:49:52.148 INFO:tasks.ceph:Creating RBD pool 2024-09-16T03:49:52.148 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T03:49:52.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.244+0000 7ff9cb05e640 10 monclient: get_monmap_and_config 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.244+0000 7ff9cb05e640 10 monclient: build_initial_monmap 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.244+0000 7ff9cb05e640 10 monclient: monmap: 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:52.245137+0000 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:52.245137+0000 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:52.247 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.245+0000 7ff9cb05e640 1 Processor -- start 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.245+0000 7ff9cb05e640 1 -- start start 2024-09-16T03:49:52.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.245+0000 7ff9cb05e640 10 monclient: init 2024-09-16T03:49:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T03:49:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9c40b5470 0x7ff9c40b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c40b5470 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9c413cbd0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 0x7ff9c4143750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c413d020 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c40ac040 con 0x7ff9c413cbd0 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c413d450 con 0x7ff9c40b5470 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c40ac2b0 con 0x7ff9c413d020 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9cb05e640 10 monclient(hunting): authenticate will time out at 750.907109s 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9c40b5470 0x7ff9c40b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9c40b5470 0x7ff9c40b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41382/0 (socket says 172.21.15.70:41382) 2024-09-16T03:49:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 0x7ff9c4143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 -- 172.21.15.70:0/2663329003 learned_addr learned my addr 172.21.15.70:0/2663329003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): get_auth_request con 0x7ff9c40b5470 auth_method 0 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c3fff640 1 --1- 172.21.15.70:0/2663329003 >> v1:172.21.15.70:6789/0 conn(0x7ff9c413cbd0 0x7ff9c4143040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59296/0 (socket says 172.21.15.70:59296) 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): get_auth_request con 0x7ff9c413d020 auth_method 0 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 -- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 msgr2=0x7ff9c4143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 0x7ff9c4143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 -- 172.21.15.70:0/2663329003 >> v1:172.21.15.70:6789/0 conn(0x7ff9c413cbd0 legacy=0x7ff9c4143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient: _finish_hunting 0 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 monclient: found mon.noname-a 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c37fe640 1 -- 172.21.15.70:0/2663329003 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 621560119 0 0) 0x7ff9c40ac040 con 0x7ff9c413cbd0 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c8dfa640 1 -- 172.21.15.70:0/2663329003 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c40aaef0 con 0x7ff9c40b5470 2024-09-16T03:49:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c95fb640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 0x7ff9c4143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.246+0000 7ff9c37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c8dfa640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c40b5850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009870 tx=0x7ff9b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b8e549d8c833e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 1 -- 172.21.15.70:0/2663329003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8037070 con 0x7ff9c40b5470 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: dump: 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.260 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 1 -- 172.21.15.70:0/2663329003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b8032440 con 0x7ff9c40b5470 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 5 monclient: authenticate success, global_id 4263 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 4 set_mon_vals no callback set 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 1 -- 172.21.15.70:0/2663329003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8032810 con 0x7ff9c40b5470 2024-09-16T03:49:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9c37fe640 10 monclient: dump: 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 10 monclient: shutdown 2024-09-16T03:49:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 msgr2=0x7ff9c40b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c40b5850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009870 tx=0x7ff9b802a270 comp rx=0 tx=0).stop 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2663329003 shutdown_connections 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413d020 0x7ff9c4143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2663329003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c40b5850 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.247+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2663329003 >> 172.21.15.70:0/2663329003 conn(0x7ff9c40679c0 msgr2=0x7ff9c4067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2663329003 shutdown_connections 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2663329003 wait complete. 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 10 monclient: build_initial_monmap 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 10 monclient: monmap: 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:52.263 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 1 librados: starting msgr at 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 1 librados: starting objecter 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.248+0000 7ff9cb05e640 1 Processor -- start 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 -- start start 2024-09-16T03:49:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 librados: setting wanted keys 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 librados: calling monclient init 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient: init 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c40b5470 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 0x7ff9c41db7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c413cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c413d020 0x7ff9c41dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c413d020 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c40b5d90 con 0x7ff9c40b5470 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4143cf0 con 0x7ff9c413d020 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c40b5f10 con 0x7ff9c413cbd0 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9cb05e640 10 monclient(hunting): authenticate will time out at 750.910766s 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 0x7ff9c41db7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41384/0 (socket says 172.21.15.70:41384) 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): get_auth_request con 0x7ff9c413cbd0 auth_method 0 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 1 -- 172.21.15.70:0/2386999110 learned_addr learned my addr 172.21.15.70:0/2386999110 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): get_auth_request con 0x7ff9c40b5470 auth_method 0 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.249+0000 7ff9c8dfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c95fb640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c413d020 0x7ff9c41dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c95fb640 10 monclient(hunting): get_auth_request con 0x7ff9c413d020 auth_method 0 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c95fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c95fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c95fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 -- 172.21.15.70:0/2386999110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c413d020 msgr2=0x7ff9c41dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c413d020 0x7ff9c41dbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 -- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 msgr2=0x7ff9c41400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 monclient: found mon.noname-c 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 -- 172.21.15.70:0/2386999110 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c41348f0 con 0x7ff9c413cbd0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c8dfa640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c3fff640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 0x7ff9c41db7f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c450 tx=0x7ff9b000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 client.?.objecter ms_handle_connect 0x7ff9c413cbd0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0018070 con 0x7ff9c413cbd0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: dump: 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: _finish_auth 0 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:49:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 5 monclient: authenticate success, global_id 4229 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b00074e0 con 0x7ff9c413cbd0 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0004a20 con 0x7ff9c413cbd0 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c17fa640 10 monclient: dump: 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9c2ffd640 4 set_mon_vals no callback set 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 10 monclient: _renew_subs 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c4134bd0 con 0x7ff9c413cbd0 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 10 monclient: _renew_subs 2024-09-16T03:49:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c413c600 con 0x7ff9c413cbd0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.250+0000 7ff9cb05e640 1 librados: init done 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.251+0000 7ff9cb05e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.251+0000 7ff9cb05e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.251+0000 7ff9cb05e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.251+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c40b5850 con 0x7ff9c413cbd0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.251+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7ff9b0004bc0 con 0x7ff9c413cbd0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 1 --2- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 0x7ff9a80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c8dfa640 1 --2- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 0x7ff9a80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 3737+0+0 (secure 0 0 0) 0x7ff9b0089da0 con 0x7ff9c413cbd0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 10 client.4229.objecter ms_dispatch 0x7ff9c4000b90 osd_map(17..17 src has 1..17) 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 3 client.4229.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T03:49:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c8dfa640 10 monclient: get_auth_request con 0x7ff9a806dc00 auth_method 0 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 3 client.4229.objecter handle_osd_map decoding full epoch 17 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 20 client.4229.objecter dump_active .. 0 homeless 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c17fa640 10 client.4229.objecter ms_handle_connect 0x7ff9a806dc00 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.252+0000 7ff9c8dfa640 1 --2- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 0x7ff9a80700c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9b802a2b0 tx=0x7ff9b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.256+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff9b0058700 con 0x7ff9c413cbd0 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.256+0000 7ff9c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.256+0000 7ff9c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.422+0000 7ff9cb05e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:49:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.422+0000 7ff9cb05e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:49:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.422+0000 7ff9cb05e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.422+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7ff9c4072230 con 0x7ff9c413cbd0 2024-09-16T03:49:52.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.842+0000 7ff9c17fa640 1 -- 172.21.15.70:0/2386999110 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) ==== 109+0+0 (secure 0 0 0) 0x7ff9b005ca00 con 0x7ff9c413cbd0 2024-09-16T03:49:52.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.842+0000 7ff9c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:49:52.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.842+0000 7ff9c17fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T03:49:52.844 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'rbd' created 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 10 librados: watch_flush enter 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 10 librados: watch_flush exit 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 20 client.4229.objecter shutdown clearing up homeless session... 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 10 client.4229.objecter successfully canceled tick 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 msgr2=0x7ff9a80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 0x7ff9a80700c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9b802a2b0 tx=0x7ff9b802e000 comp rx=0 tx=0).stop 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 10 monclient: shutdown 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 msgr2=0x7ff9c41db7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 0x7ff9c41db7f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c450 tx=0x7ff9b000c910 comp rx=0 tx=0).stop 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 shutdown_connections 2024-09-16T03:49:52.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9a806dc00 0x7ff9a80700c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9c413d020 0x7ff9c41dbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9c413cbd0 0x7ff9c41db7f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 --2- 172.21.15.70:0/2386999110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9c40b5470 0x7ff9c41400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 >> 172.21.15.70:0/2386999110 conn(0x7ff9c40679c0 msgr2=0x7ff9c4067da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 shutdown_connections 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 -- 172.21.15.70:0/2386999110 wait complete. 2024-09-16T03:49:52.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.844+0000 7ff9cb05e640 1 librados: shutdown 2024-09-16T03:49:52.856 DEBUG:teuthology.orchestra.run.smithi070:> rbd --cluster ceph pool init rbd 2024-09-16T03:49:52.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 10 monclient: get_monmap_and_config 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 10 monclient: build_initial_monmap 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 10 monclient: monmap: 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:52.974666+0000 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:52.974666+0000 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:52.976 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:52.977 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:52.977 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 1 Processor -- start 2024-09-16T03:49:52.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 1 -- start start 2024-09-16T03:49:52.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.974+0000 7f816c6e8d80 10 monclient: init 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient: _reopen_session rank -1 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 0x56524e9ec140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-b con 0x56524e92ec50 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-c-legacy con 0x56524e953a50 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-a-legacy con 0x56524e95ca40 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56524e7b2050 con 0x56524e95ca40 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56524e79bce0 con 0x56524e953a50 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x56524e7c0930 con 0x56524e92ec50 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): _renew_subs 2024-09-16T03:49:52.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c6e8d80 10 monclient(hunting): authenticate will time out at 751.636549s 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 0x56524e9ec140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c442640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x56524e95ca40 0x56524e95b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59302/0 (socket says 172.21.15.70:59302) 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): get_auth_request con 0x56524e92ec50 auth_method 0 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816c442640 1 -- 172.21.15.70:0/685952800 learned_addr learned my addr 172.21.15.70:0/685952800 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 1 -- 172.21.15.70:0/685952800 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498571483 0 0) 0x56524e7b2050 con 0x56524e95ca40 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 10 monclient(hunting): _negotiate 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 10 monclient(hunting): my global_id is 4272 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816a1de640 1 -- 172.21.15.70:0/685952800 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f814c001e10 con 0x56524e95ca40 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.975+0000 7f816b1e0640 1 -- 172.21.15.70:0/685952800 >> v1:172.21.15.163:6789/0 conn(0x56524e953a50 legacy=0x56524e95c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 1 -- 172.21.15.70:0/685952800 >> v1:172.21.15.70:6789/0 conn(0x56524e95ca40 legacy=0x56524e95b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 1 monclient: found mon.noname-b 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.976+0000 7f816b1e0640 1 -- 172.21.15.70:0/685952800 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56524e9ec860 con 0x56524e92ec50 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816b1e0640 1 --2- 172.21.15.70:0/685952800 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 0x56524e9ec140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f815400a4a0 tx=0x7f8154045c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a25ff5390a3bc139 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 1 -- 172.21.15.70:0/685952800 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154058600 con 0x56524e92ec50 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: dump: 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: _finish_auth 0 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: _check_auth_tickets 2024-09-16T03:49:52.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816c6e8d80 5 monclient: authenticate success, global_id 4232 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816c6e8d80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 1 -- 172.21.15.70:0/685952800 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81540587a0 con 0x56524e92ec50 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816c6e8d80 10 monclient: get_monmap_and_config success 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816c6e8d80 4 set_mon_vals no callback set 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 1 -- 172.21.15.70:0/685952800 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154058b50 con 0x56524e92ec50 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.977+0000 7f816a1de640 10 monclient: dump: 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.985 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 10 monclient: shutdown 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 -- 172.21.15.70:0/685952800 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 msgr2=0x56524e9ec140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/685952800 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 0x56524e9ec140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f815400a4a0 tx=0x7f8154045c40 comp rx=0 tx=0).stop 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 -- 172.21.15.70:0/685952800 shutdown_connections 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/685952800 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e92ec50 0x56524e9ec140 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 -- 172.21.15.70:0/685952800 >> 172.21.15.70:0/685952800 conn(0x56524e961220 msgr2=0x56524e95d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 -- 172.21.15.70:0/685952800 shutdown_connections 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 1 -- 172.21.15.70:0/685952800 wait complete. 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 10 monclient: build_initial_monmap 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.978+0000 7f816c6e8d80 10 monclient: monmap: 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 librados: starting msgr at 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 librados: starting objecter 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 Processor -- start 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 -- start start 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 librados: setting wanted keys 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 librados: calling monclient init 2024-09-16T03:49:52.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient: init 2024-09-16T03:49:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient: _reopen_session rank -1 2024-09-16T03:49:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56524e92ec50 0x56524e957050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-b con 0x56524e92ec50 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:52.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56524e953a50 0x56524e9fad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-a con 0x56524e953a50 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 0x56524e9fe250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): picked mon.noname-c con 0x56524e95ca40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56524e7c2540 con 0x56524e953a50 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56524e81bc80 con 0x56524e92ec50 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x56524e95bb30 con 0x56524e95ca40 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): _renew_subs 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c6e8d80 10 monclient(hunting): authenticate will time out at 751.640895s 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816a9df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56524e953a50 0x56524e9fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.979+0000 7f816c442640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 0x56524e9fe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 0x56524e9fe250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46556/0 (socket says 172.21.15.70:46556) 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 -- 172.21.15.70:0/403231334 learned_addr learned my addr 172.21.15.70:0/403231334 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): get_auth_request con 0x56524e953a50 auth_method 0 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): get_auth_request con 0x56524e95ca40 auth_method 0 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56524e92ec50 0x56524e957050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:52.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): get_auth_request con 0x56524e92ec50 auth_method 0 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816b1e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-16T03:49:52.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 -- 172.21.15.70:0/403231334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56524e92ec50 msgr2=0x56524e957050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56524e92ec50 0x56524e957050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 -- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56524e953a50 msgr2=0x56524e9fad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56524e953a50 0x56524e9fad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient: _finish_hunting 0 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 monclient: found mon.noname-c 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56524e958230 con 0x56524e95ca40 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816a9df640 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56524e953a50 0x56524e9fad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c442640 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 0x56524e9fe250 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f815c00bbc0 tx=0x7f815c009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 client.?.objecter ms_handle_connect 0x56524e95ca40 2024-09-16T03:49:52.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c012070 con 0x56524e95ca40 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: dump: 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.997 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f815c00e690 con 0x56524e95ca40 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 5 monclient: authenticate success, global_id 4235 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 monclient: _renew_subs 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x56524e958510 con 0x56524e95ca40 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815c00eac0 con 0x56524e95ca40 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:52.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81637fe640 10 monclient: dump: 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f81699dd640 4 set_mon_vals no callback set 2024-09-16T03:49:52.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 monclient: _renew_subs 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x56524e95a130 con 0x56524e95ca40 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 1 librados: init done 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.980+0000 7f816c6e8d80 10 librados: wait_for_osdmap waiting 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.981+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f815c015070 con 0x56524e95ca40 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.70:6804/198622594 conn(0x7f815006bd30 0x7f815006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816b1e0640 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.70:6804/198622594 conn(0x7f815006bd30 0x7f815006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4079+0+0 (secure 0 0 0) 0x7f815c0899b0 con 0x56524e95ca40 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_map(18..18 src has 1..18) 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 3 client.4235.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 3 client.4235.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:49:53.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816b1e0640 10 monclient: get_auth_request con 0x7f815006bd30 auth_method 0 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816c6e8d80 10 librados: wait_for_osdmap done waiting 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816c6e8d80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816c6e8d80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816c6e8d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x56524e956150 con 0x56524e95ca40 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f81637fe640 10 client.4235.objecter ms_handle_connect 0x7f815006bd30 2024-09-16T03:49:53.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:52.982+0000 7f816b1e0640 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.70:6804/198622594 conn(0x7f815006bd30 0x7f815006e1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8154057000 tx=0x7f8154005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:53.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.844+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v19) ==== 141+0+0 (secure 0 0 0) 0x7f815c0581c0 con 0x56524e95ca40 2024-09-16T03:49:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.844+0000 7f81637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:49:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.844+0000 7f81637fe640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T03:49:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.844+0000 7f816c6e8d80 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.844+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81540587a0 con 0x56524e95ca40 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f815c01c0a0 con 0x56524e95ca40 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81699dd640 10 client.4235.objecter _get_latest_version latest 19, waiting 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81699dd640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81699dd640 10 monclient: _renew_subs 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81699dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81699dd640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=19}) -- 0x7f8144003c90 con 0x56524e95ca40 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 8 ==== osd_map(19..19 src has 1..19) ==== 666+0+0 (secure 0 0 0) 0x7f815c088650 con 0x56524e95ca40 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_map(19..19 src has 1..19) 2024-09-16T03:49:53.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 3 client.4235.objecter handle_osd_map got epochs [19,19] > 18 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 3 client.4235.objecter handle_osd_map decoding incremental epoch 19 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.848+0000 7f81637fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 10 librados: create oid=rbd_trash nspace= 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 10 client.4235.objecter _op_submit op 0x56524e9febb0 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 20 client.4235.objecter _calc_target epoch 19 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 20 client.4235.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 10 client.4235.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.163:6808/1625318059 conn(0x56524e9e7b10 0x56524e9e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 20 client.4235.objecter _get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 10 client.4235.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 20 client.4235.objecter get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 15 client.4235.objecter _session_op_assign 5 1 2024-09-16T03:49:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 15 client.4235.objecter _send_op 1 to 2.3 on osd.5 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> v2:172.21.15.163:6808/1625318059 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x56524ea07830 con 0x56524e9e7b10 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816c6e8d80 5 client.4235.objecter 1 in flight 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816a9df640 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.163:6808/1625318059 conn(0x56524e9e7b10 0x56524e9e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.849+0000 7f816a9df640 10 monclient: get_auth_request con 0x56524e9e7b10 auth_method 0 2024-09-16T03:49:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.850+0000 7f81637fe640 10 client.4235.objecter ms_handle_connect 0x56524e9e7b10 2024-09-16T03:49:53.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.850+0000 7f816a9df640 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.163:6808/1625318059 conn(0x56524e9e7b10 0x56524e9e7ef0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:53.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 1 -- 172.21.15.70:0/403231334 <== osd.5 v2:172.21.15.163:6808/1625318059 1 ==== osd_op_reply(1 rbd_trash [create] v19'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f815800ab10 con 0x56524e9e7b10 2024-09-16T03:49:53.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_op_reply(1 rbd_trash [create] v19'1 uv1 ondisk = 0) 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 10 client.4235.objecter in handle_osd_op_reply 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 7 client.4235.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 15 client.4235.objecter handle_osd_op_reply completed tid 1 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 15 client.4235.objecter _finish_op 1 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 15 client.4235.objecter _session_op_remove 5 1 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816a9df640 5 client.4235.objecter 0 in flight 2024-09-16T03:49:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.856+0000 7f816c6e8d80 10 librados: Objecter returned from create r=0 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 10 client.4235.objecter _op_submit op 0x56524ea58fc0 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 20 client.4235.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 20 client.4235.objecter _get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 20 client.4235.objecter get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 15 client.4235.objecter _session_op_assign 5 2 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 15 client.4235.objecter _send_op 2 to 2.2 on osd.5 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> v2:172.21.15.163:6808/1625318059 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x56524ea59510 con 0x56524e9e7b10 2024-09-16T03:49:53.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.858+0000 7f816c6e8d80 5 client.4235.objecter 1 in flight 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 1 -- 172.21.15.70:0/403231334 <== osd.5 v2:172.21.15.163:6808/1625318059 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f815800ab10 con 0x56524e9e7b10 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 10 client.4235.objecter in handle_osd_op_reply 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 7 client.4235.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 10 client.4235.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 15 client.4235.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 15 client.4235.objecter _finish_op 2 2024-09-16T03:49:53.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 15 client.4235.objecter _session_op_remove 5 2 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816a9df640 5 client.4235.objecter 0 in flight 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 10 client.4235.objecter _op_submit op 0x56524e9525c0 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 20 client.4235.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 20 client.4235.objecter _get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 20 client.4235.objecter get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 15 client.4235.objecter _session_op_assign 5 3 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 15 client.4235.objecter _send_op 3 to 2.2 on osd.5 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 --> v2:172.21.15.163:6808/1625318059 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) -- 0x56524e952a70 con 0x56524e9e7b10 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.859+0000 7f816c6e8d80 5 client.4235.objecter 1 in flight 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 1 -- 172.21.15.70:0/403231334 <== osd.5 v2:172.21.15.163:6808/1625318059 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f815800ab10 con 0x56524e9e7b10 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 10 client.4235.objecter in handle_osd_op_reply 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 7 client.4235.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 10 client.4235.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 15 client.4235.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:49:53.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 15 client.4235.objecter _finish_op 3 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 15 client.4235.objecter _session_op_remove 5 3 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f816a9df640 5 client.4235.objecter 0 in flight 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f81691dc640 10 client.4235.objecter allocate_selfmanaged_snap; pool: 2 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f81691dc640 10 client.4235.objecter pool_op_submit 4 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f81691dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:53.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:53.860+0000 7f81691dc640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f81480030d0 con 0x56524e95ca40 2024-09-16T03:49:54.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v20) ==== 55+0+0 (secure 0 0 0) 0x7f815c086080 con 0x56524e95ca40 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 pool_op_reply(tid 4 (0) Success v20) 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v20) 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter have request 4 at 0x7f8148003060 Op: create unmanaged snap 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 20 client.4235.objecter waiting for client to reach epoch 20 before calling back 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 monclient: _renew_subs 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=20}) -- 0x7f8150072020 con 0x56524e95ca40 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter done 2024-09-16T03:49:54.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 10 ==== osd_map(20..20 src has 1..20) ==== 626+0+0 (secure 0 0 0) 0x7f815c056260 con 0x56524e95ca40 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_map(20..20 src has 1..20) 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 3 client.4235.objecter handle_osd_map got epochs [20,20] > 19 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 3 client.4235.objecter handle_osd_map decoding incremental epoch 20 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81637fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 librados: queue_aio_write 0x56524ea54550 completion 0x56524e7dbd80 write_seq 1 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 10 client.4235.objecter _op_submit op 0x7f8144006380 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 client.4235.objecter _calc_target epoch 20 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:49:54.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 client.4235.objecter _get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 client.4235.objecter get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 15 client.4235.objecter _session_op_assign 5 5 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 15 client.4235.objecter _send_op 5 to 2.2 on osd.5 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 1 -- 172.21.15.70:0/403231334 --> v2:172.21.15.163:6808/1625318059 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x7f8144006830 con 0x56524e9e7b10 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:54.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.852+0000 7f81699dd640 5 client.4235.objecter 1 in flight 2024-09-16T03:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 1 -- 172.21.15.70:0/403231334 <== osd.5 v2:172.21.15.163:6808/1625318059 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v20'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f815800ab10 con 0x56524e9e7b10 2024-09-16T03:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_op_reply(5 rbd_info [create,write 0~8] v20'1 uv1 ondisk = 0) 2024-09-16T03:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 10 client.4235.objecter in handle_osd_op_reply 2024-09-16T03:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 7 client.4235.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-16T03:49:54.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 10 client.4235.objecter op 1 rval 0 len 0 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 15 client.4235.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 15 client.4235.objecter _finish_op 5 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 15 client.4235.objecter _session_op_remove 5 5 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 5 client.4235.objecter 0 in flight 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f816a9df640 20 librados: complete_aio_write 0x56524e7dbd80 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f81691dc640 10 client.4235.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f81691dc640 10 client.4235.objecter pool_op_submit 6 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f81691dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:54.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:54.856+0000 7f81691dc640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v20) -- 0x7f8148004520 con 0x56524e95ca40 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v21) ==== 43+0+0 (secure 0 0 0) 0x7f815c05c4c0 con 0x56524e95ca40 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 pool_op_reply(tid 6 (0) Success v21) 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v21) 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter have request 6 at 0x7f8148003060 Op: delete unmanaged snap 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 20 client.4235.objecter waiting for client to reach epoch 21 before calling back 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 monclient: _renew_subs 2024-09-16T03:49:55.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=21}) -- 0x7f81500721f0 con 0x56524e95ca40 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter done 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 1 -- 172.21.15.70:0/403231334 <== mon.2 v2:172.21.15.70:3301/0 12 ==== osd_map(21..21 src has 1..21) ==== 654+0+0 (secure 0 0 0) 0x7f815c0564c0 con 0x56524e95ca40 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_map(21..21 src has 1..21) 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 3 client.4235.objecter handle_osd_map got epochs [21,21] > 20 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 3 client.4235.objecter handle_osd_map decoding incremental epoch 21 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.855+0000 7f81637fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 librados: queue_aio_write 0x56524ea54550 completion 0x7f8148001d60 write_seq 2 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 10 client.4235.objecter _op_submit op 0x7f8144006380 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 client.4235.objecter _calc_target epoch 21 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 client.4235.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 10 client.4235.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:49:55.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 client.4235.objecter _get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 10 client.4235.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 client.4235.objecter get_session s=0x56524e9ff060 osd=5 3 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 15 client.4235.objecter _session_op_assign 5 7 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 15 client.4235.objecter _send_op 7 to 2.2 on osd.5 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 1 -- 172.21.15.70:0/403231334 --> v2:172.21.15.163:6808/1625318059 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x7f8144006bd0 con 0x56524e9e7b10 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:55.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.856+0000 7f81699dd640 5 client.4235.objecter 1 in flight 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 1 -- 172.21.15.70:0/403231334 <== osd.5 v2:172.21.15.163:6808/1625318059 5 ==== osd_op_reply(7 rbd_info [write 0~19] v21'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f815800ab10 con 0x56524e9e7b10 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 10 client.4235.objecter ms_dispatch 0x56524e832c20 osd_op_reply(7 rbd_info [write 0~19] v21'2 uv2 ondisk = 0) 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 10 client.4235.objecter in handle_osd_op_reply 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 7 client.4235.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 10 client.4235.objecter op 0 rval 0 len 0 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 15 client.4235.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 15 client.4235.objecter _finish_op 7 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 4 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 15 client.4235.objecter _session_op_remove 5 7 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 5 client.4235.objecter 0 in flight 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816a9df640 20 librados: complete_aio_write 0x7f8148001d60 2024-09-16T03:49:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 10 librados: watch_flush enter 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 10 librados: watch_flush exit 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 10 client.4235.objecter close_session for osd.5 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 >> v2:172.21.15.163:6808/1625318059 conn(0x56524e9e7b10 msgr2=0x56524e9e7ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.163:6808/1625318059 conn(0x56524e9e7b10 0x56524e9e7ef0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 20 client.4235.objecter put_session s=0x56524e9ff060 osd=5 1 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 20 client.4235.objecter shutdown clearing up homeless session... 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 10 client.4235.objecter successfully canceled tick 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 >> v2:172.21.15.70:6804/198622594 conn(0x7f815006bd30 msgr2=0x7f815006e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/403231334 >> v2:172.21.15.70:6804/198622594 conn(0x7f815006bd30 0x7f815006e1f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8154057000 tx=0x7f8154005f90 comp rx=0 tx=0).stop 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 10 monclient: shutdown 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 msgr2=0x56524e9fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 --2- 172.21.15.70:0/403231334 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x56524e95ca40 0x56524e9fe250 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f815c00bbc0 tx=0x7f815c009640 comp rx=0 tx=0).stop 2024-09-16T03:49:55.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c442640 1 -- 172.21.15.70:0/403231334 reap_dead start 2024-09-16T03:49:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 shutdown_connections 2024-09-16T03:49:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 >> 172.21.15.70:0/403231334 conn(0x56524e961220 msgr2=0x56524e95b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 shutdown_connections 2024-09-16T03:49:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.861+0000 7f816c6e8d80 1 -- 172.21.15.70:0/403231334 wait complete. 2024-09-16T03:49:55.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.862+0000 7f816c6e8d80 1 librados: shutdown 2024-09-16T03:49:55.868 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T03:49:55.869 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T03:49:55.869 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T03:49:55.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T03:49:55.870 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.972+0000 7f5748154640 10 monclient: get_monmap_and_config 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.972+0000 7f5748154640 10 monclient: build_initial_monmap 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.972+0000 7f5748154640 10 monclient: monmap: 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:55.974184+0000 2024-09-16T03:49:55.975 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:55.974184+0000 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:55.976 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:55.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.973+0000 7f5748154640 1 Processor -- start 2024-09-16T03:49:55.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.973+0000 7f5748154640 1 -- start start 2024-09-16T03:49:55.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.973+0000 7f5748154640 10 monclient: init 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f574006c100 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57400af520 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f57400b0060 0x7f57400b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): picked mon.noname-c con 0x7f57400b0060 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57400aebb0 con 0x7f574006c100 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57400aee20 con 0x7f57400af520 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f574006c4e0 con 0x7f57400b0060 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 10 monclient(hunting): authenticate will time out at 754.636245s 2024-09-16T03:49:55.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5745ef0640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f574006c100 0x7f57400b1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33668/0 (socket says 172.21.15.70:33668) 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5745ef0640 1 -- 172.21.15.70:0/1129164847 learned_addr learned my addr 172.21.15.70:0/1129164847 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863566355 0 0) 0x7f57400aebb0 con 0x7f574006c100 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): _negotiate 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): my global_id is 4278 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f572c003520 con 0x7f574006c100 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 1 --2- 172.21.15.70:0/1129164847 >> v2:172.21.15.163:3300/0 conn(0x7f57400b0060 0x7f57400b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): get_auth_request con 0x7f57400b0060 auth_method 0 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:55.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 361840299 0 0) 0x7f572c003520 con 0x7f574006c100 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 >> v2:172.21.15.163:3300/0 conn(0x7f57400b0060 msgr2=0x7f57400b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 --2- 172.21.15.70:0/1129164847 >> v2:172.21.15.163:3300/0 conn(0x7f57400b0060 0x7f57400b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 >> v1:172.21.15.163:6789/0 conn(0x7f57400af520 legacy=0x7f57400af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _finish_hunting 0 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57400af100 con 0x7f574006c100 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _finish_auth 0 2024-09-16T03:49:55.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _check_auth_tickets 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f5730001670 con 0x7f574006c100 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 5 monclient: authenticate success, global_id 4278 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5748154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: dump: 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 >> v1:172.21.15.70:6789/0 conn(0x7f574006c100 legacy=0x7f57400b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f572c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): picked mon.a con 0x7f572c0039b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572c004800 0x7f572c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): picked mon.b con 0x7f572c004800 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f572c0051c0 0x7f572c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): picked mon.c con 0x7f572c0051c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f574006c4e0 con 0x7f572c0039b0 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f572c00c160 con 0x7f572c004800 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f572c00c360 con 0x7f572c0051c0 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.974+0000 7f5744eee640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5744eee640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5748154640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f572c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): get_auth_request con 0x7f572c0039b0 auth_method 0 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f572c0051c0 0x7f572c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572c004800 0x7f572c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): get_auth_request con 0x7f572c0051c0 auth_method 0 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): get_auth_request con 0x7f572c004800 auth_method 0 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient(hunting): handle_auth_done global_id 4278 payload 293 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 -- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f572c0051c0 msgr2=0x7f572c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f572c0051c0 0x7f572c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 -- 172.21.15.70:0/1129164847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572c004800 msgr2=0x7f572c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f572c004800 0x7f572c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:55.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 monclient: found mon.a 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f57466f1640 1 -- 172.21.15.70:0/1129164847 reap_dead start 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 1 -- 172.21.15.70:0/1129164847 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c003520 con 0x7f572c0039b0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient: _finish_auth 0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 10 monclient: _check_auth_tickets 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.975+0000 7f5745ef0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5745ef0640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f572c003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f573000c240 tx=0x7f572c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9709848f87982673 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730007d10 con 0x7f572c0039b0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: dump: 2024-09-16T03:49:55.995 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:55.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5730005ad0 con 0x7f572c0039b0 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 1 -- 172.21.15.70:0/1129164847 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730007070 con 0x7f572c0039b0 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 4 set_mon_vals no callback set 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5744eee640 10 monclient: dump: 2024-09-16T03:49:55.997 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 10 monclient: shutdown 2024-09-16T03:49:55.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 -- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 msgr2=0x7f572c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f572c003dd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f573000c240 tx=0x7f572c005830 comp rx=0 tx=0).stop 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 -- 172.21.15.70:0/1129164847 shutdown_connections 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 --2- 172.21.15.70:0/1129164847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f572c003dd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 -- 172.21.15.70:0/1129164847 >> 172.21.15.70:0/1129164847 conn(0x7f5740068490 msgr2=0x7f57400b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 -- 172.21.15.70:0/1129164847 shutdown_connections 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.976+0000 7f5748154640 1 -- 172.21.15.70:0/1129164847 wait complete. 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 10 monclient: build_initial_monmap 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 10 monclient: monmap: 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:55.999 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 librados: starting msgr at 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 librados: starting objecter 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 Processor -- start 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 -- start start 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 librados: setting wanted keys 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 1 librados: calling monclient init 2024-09-16T03:49:56.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.977+0000 7f5748154640 10 monclient: init 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): picked mon.noname-a con 0x7f572c0039b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 0x7f574013f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): picked mon.noname-c con 0x7f5740120980 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574013fbc0 0x7f57401d80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): picked mon.noname-b con 0x7f574013fbc0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f574006cdc0 con 0x7f572c0039b0 2024-09-16T03:49:56.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f574006c940 con 0x7f574013fbc0 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f574006d150 con 0x7f5740120980 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5748154640 10 monclient(hunting): authenticate will time out at 754.640461s 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 0x7f574013f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48962/0 (socket says 172.21.15.70:48962) 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 1 -- 172.21.15.70:0/3210720990 learned_addr learned my addr 172.21.15.70:0/3210720990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:56.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): get_auth_request con 0x7f572c0039b0 auth_method 0 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): get_auth_request con 0x7f5740120980 auth_method 0 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57466f1640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574013fbc0 0x7f57401d80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57466f1640 10 monclient(hunting): get_auth_request con 0x7f574013fbc0 auth_method 0 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57466f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57466f1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57466f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 -- 172.21.15.70:0/3210720990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574013fbc0 msgr2=0x7f57401d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574013fbc0 0x7f57401d80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 -- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 msgr2=0x7f5740121170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient: _finish_hunting 0 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 monclient: found mon.noname-c 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:56.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f57456ef640 1 -- 172.21.15.70:0/3210720990 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5740122a90 con 0x7f5740120980 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.978+0000 7f5745ef0640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57456ef640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 0x7f574013f680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f573c00f580 tx=0x7f573c00fa40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 client.?.objecter ms_handle_connect 0x7f5740120980 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c0053a0 con 0x7f5740120980 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: dump: 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.005 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:56.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573c018070 con 0x7f5740120980 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 5 monclient: authenticate success, global_id 4238 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c003830 con 0x7f5740120980 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f57367fc640 10 monclient: dump: 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.007 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 10 monclient: _renew_subs 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5737fff640 4 set_mon_vals no callback set 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5740121840 con 0x7f5740120980 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 10 monclient: _renew_subs 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5740140020 con 0x7f5740120980 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.979+0000 7f5748154640 1 librados: init done 2024-09-16T03:49:56.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.980+0000 7f5748154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.980+0000 7f5748154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.980+0000 7f5748154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.980+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57400b0540 con 0x7f5740120980 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.980+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f573c0039d0 con 0x7f5740120980 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 1 --2- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 0x7f57240700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f5745ef0640 1 --2- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 0x7f57240700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f573c08ab30 con 0x7f5740120980 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 10 client.4238.objecter ms_dispatch 0x7f5740000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 3 client.4238.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 3 client.4238.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f5745ef0640 10 monclient: get_auth_request con 0x7f572406dc30 auth_method 0 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f57367fc640 10 client.4238.objecter ms_handle_connect 0x7f572406dc30 2024-09-16T03:49:56.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.981+0000 7f5745ef0640 1 --2- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 0x7f57240700f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5730000f60 tx=0x7f573002d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.986+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f57400aee20 con 0x7f5740120980 2024-09-16T03:49:56.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.986+0000 7f57367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:55.986+0000 7f57367fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:56.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.198+0000 7f5748154640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:49:56.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.198+0000 7f5748154640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:49:56.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.198+0000 7f5748154640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.198+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f574006de20 con 0x7f5740120980 2024-09-16T03:49:56.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.201+0000 7f57367fc640 1 -- 172.21.15.70:0/3210720990 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173442 (secure 0 0 0) 0x7f573c0598c0 con 0x7f5740120980 2024-09-16T03:49:56.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.201+0000 7f57367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:49:56.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.201+0000 7f57367fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:56.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 10 librados: watch_flush enter 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 10 librados: watch_flush exit 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 20 client.4238.objecter shutdown clearing up homeless session... 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 10 client.4238.objecter successfully canceled tick 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 msgr2=0x7f57240700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 0x7f57240700f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5730000f60 tx=0x7f573002d000 comp rx=0 tx=0).stop 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 10 monclient: shutdown 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 msgr2=0x7f574013f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 0x7f574013f680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f573c00f580 tx=0x7f573c00fa40 comp rx=0 tx=0).stop 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 shutdown_connections 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> v2:172.21.15.70:6804/198622594 conn(0x7f572406dc30 0x7f57240700f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f574013fbc0 0x7f57401d80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5740120980 0x7f574013f680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 --2- 172.21.15.70:0/3210720990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f572c0039b0 0x7f5740121170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.203+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 >> 172.21.15.70:0/3210720990 conn(0x7f5740068490 msgr2=0x7f57400b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.204+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 shutdown_connections 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.204+0000 7f5748154640 1 -- 172.21.15.70:0/3210720990 wait complete. 2024-09-16T03:49:56.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.204+0000 7f5748154640 1 librados: shutdown 2024-09-16T03:49:56.216 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":198622594}]},"active_addr":"172.21.15.70:6804/198622594","active_change":"2024-09-16T03:49:38.626256+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1757558108}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":460033411}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1768628161}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1480501704}]}]} 2024-09-16T03:49:56.220 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T03:49:56.221 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:49:56.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.351+0000 7f13dbd4b640 10 monclient: get_monmap_and_config 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.351+0000 7f13dbd4b640 10 monclient: build_initial_monmap 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.351+0000 7f13dbd4b640 10 monclient: monmap: 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:56.353344+0000 2024-09-16T03:49:56.354 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:56.353344+0000 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:56.355 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.352+0000 7f13dbd4b640 1 Processor -- start 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.352+0000 7f13dbd4b640 1 -- start start 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.352+0000 7f13dbd4b640 10 monclient: init 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-b con 0x7f13d40ade90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 0x7f13d40ab740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-a con 0x7f13d413c900 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13d413cce0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40ac390 con 0x7f13d413cce0 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f13d40ae270 con 0x7f13d413c900 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f13d40ad420 con 0x7f13d40ade90 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13dbd4b640 10 monclient(hunting): authenticate will time out at 755.015258s 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 0x7f13d40ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45598/0 (socket says 172.21.15.70:45598) 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 -- 172.21.15.70:0/3972987020 learned_addr learned my addr 172.21.15.70:0/3972987020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): get_auth_request con 0x7f13d40ade90 auth_method 0 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): get_auth_request con 0x7f13d413c900 auth_method 0 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 -- 172.21.15.70:0/3972987020 >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 msgr2=0x7f13d40ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 --2- 172.21.15.70:0/3972987020 >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 0x7f13d40ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 -- 172.21.15.70:0/3972987020 >> v1:172.21.15.163:6789/0 conn(0x7f13d413cce0 legacy=0x7f13d40abc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient: _finish_hunting 0 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 monclient: found mon.noname-b 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d9ae7640 1 -- 172.21.15.70:0/3972987020 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d40ad700 con 0x7f13d40ade90 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.353+0000 7f13d92e6640 1 --2- 172.21.15.70:0/3972987020 >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 0x7f13d40ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d9ae7640 1 --2- 172.21.15.70:0/3972987020 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb1cf5f88869ed9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 1 -- 172.21.15.70:0/3972987020 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0037070 con 0x7f13d40ade90 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: dump: 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: _finish_auth 0 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: _check_auth_tickets 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13dbd4b640 5 monclient: authenticate success, global_id 4241 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13dbd4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 1 -- 172.21.15.70:0/3972987020 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0032440 con 0x7f13d40ade90 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13dbd4b640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13dbd4b640 4 set_mon_vals no callback set 2024-09-16T03:49:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 1 -- 172.21.15.70:0/3972987020 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0032810 con 0x7f13d40ade90 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.354+0000 7f13d8ae5640 10 monclient: dump: 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.372 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 10 monclient: shutdown 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 -- 172.21.15.70:0/3972987020 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 msgr2=0x7f13d413c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/3972987020 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d002a200 comp rx=0 tx=0).stop 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 -- 172.21.15.70:0/3972987020 shutdown_connections 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/3972987020 >> v2:172.21.15.70:3300/0 conn(0x7f13d413c900 0x7f13d40ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/3972987020 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d413c3c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 -- 172.21.15.70:0/3972987020 >> 172.21.15.70:0/3972987020 conn(0x7f13d40b1830 msgr2=0x7f13d40b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 -- 172.21.15.70:0/3972987020 shutdown_connections 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 1 -- 172.21.15.70:0/3972987020 wait complete. 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 10 monclient: build_initial_monmap 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.355+0000 7f13dbd4b640 10 monclient: monmap: 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 librados: starting msgr at 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 librados: starting objecter 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 Processor -- start 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 -- start start 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 librados: setting wanted keys 2024-09-16T03:49:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 librados: calling monclient init 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient: init 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-c con 0x7f13d40ade90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13d413c900 0x7f13d4137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-b con 0x7f13d413c900 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d413cce0 0x7f13d40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): picked mon.noname-a con 0x7f13d413cce0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13d40af0a0 con 0x7f13d413cce0 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f13d40ab240 con 0x7f13d413c900 2024-09-16T03:49:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f13d40af220 con 0x7f13d40ade90 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13dbd4b640 10 monclient(hunting): authenticate will time out at 755.018728s 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45600/0 (socket says 172.21.15.70:45600) 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 1 -- 172.21.15.70:0/4140988742 learned_addr learned my addr 172.21.15.70:0/4140988742 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): get_auth_request con 0x7f13d40ade90 auth_method 0 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d413cce0 0x7f13d40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): get_auth_request con 0x7f13d413cce0 auth_method 0 2024-09-16T03:49:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.356+0000 7f13da2e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d92e6640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13d413c900 0x7f13d4137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d92e6640 10 monclient(hunting): get_auth_request con 0x7f13d413c900 auth_method 0 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d92e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d92e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d92e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 -- 172.21.15.70:0/4140988742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13d413c900 msgr2=0x7f13d4137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13d413c900 0x7f13d4137a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 -- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d413cce0 msgr2=0x7f13d40b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d413cce0 0x7f13d40b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 10 monclient: _finish_hunting 0 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 monclient: found mon.noname-c 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 -- 172.21.15.70:0/4140988742 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d4138770 con 0x7f13d40ade90 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13d9ae7640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 client.?.objecter ms_handle_connect 0x7f13d40ade90 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0037040 con 0x7f13d40ade90 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.379 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 5 monclient: authenticate success, global_id 4244 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0042070 con 0x7f13d40ade90 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 10 monclient: _renew_subs 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d4137fa0 con 0x7f13d40ade90 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0030880 con 0x7f13d40ade90 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c67fc640 10 monclient: dump: 2024-09-16T03:49:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 10 monclient: _renew_subs 2024-09-16T03:49:56.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d40b4650 con 0x7f13d40ade90 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13c7fff640 4 set_mon_vals no callback set 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.357+0000 7f13dbd4b640 1 librados: init done 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.358+0000 7f13dbd4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.358+0000 7f13dbd4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.358+0000 7f13dbd4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.358+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d413c3c0 con 0x7f13d40ade90 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.358+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f13d002ad10 con 0x7f13d40ade90 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 1 --2- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 0x7f13b80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f13d00ab250 con 0x7f13d40ade90 2024-09-16T03:49:56.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 10 client.4244.objecter ms_dispatch 0x7f13d4000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 3 client.4244.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 3 client.4244.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13d92e6640 1 --2- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 0x7f13b80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13d92e6640 10 monclient: get_auth_request con 0x7f13b806dc00 auth_method 0 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 20 client.4244.objecter dump_active .. 0 homeless 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13c67fc640 10 client.4244.objecter ms_handle_connect 0x7f13b806dc00 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.359+0000 7f13d92e6640 1 --2- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 0x7f13b80700c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13c800a700 tx=0x7f13c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.363+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f13d007a060 con 0x7f13d40ade90 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.363+0000 7f13c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.363+0000 7f13c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.530+0000 7f13dbd4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13dbd4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13dbd4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13d4138940 con 0x7f13d40ade90 2024-09-16T03:49:56.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13c67fc640 1 -- 172.21.15.70:0/4140988742 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+12935 (secure 0 0 0) 0x7f13d00ab030 con 0x7f13d40ade90 2024-09-16T03:49:56.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.531+0000 7f13c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:56.534 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:49:56.534 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":21,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:55.850087+0000","last_up_change":"2024-09-16T03:49:46.819202+0000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:49:46.663530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:49:52.440963+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1243681362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1243681362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":1243681362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1243681362}]},"public_addr":"172.21.15.70:6800/1243681362","cluster_addr":"172.21.15.70:6801/1243681362","heartbeat_back_addr":"172.21.15.70:6803/1243681362","heartbeat_front_addr":"172.21.15.70:6802/1243681362","state":["exists","up"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":1499120614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":1499120614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1499120614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":1499120614}]},"public_addr":"172.21.15.70:6813/1499120614","cluster_addr":"172.21.15.70:6814/1499120614","heartbeat_back_addr":"172.21.15.70:6816/1499120614","heartbeat_front_addr":"172.21.15.70:6815/1499120614","state":["exists","up"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":2584301924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2584301924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2584301924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":2584301924}]},"public_addr":"172.21.15.70:6809/2584301924","cluster_addr":"172.21.15.70:6810/2584301924","heartbeat_back_addr":"172.21.15.70:6812/2584301924","heartbeat_front_addr":"172.21.15.70:6811/2584301924","state":["exists","up"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3986044637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3986044637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3986044637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3986044637}]},"public_addr":"172.21.15.70:6805/3986044637","cluster_addr":"172.21.15.70:6806/3986044637","heartbeat_back_addr":"172.21.15.70:6808/3986044637","heartbeat_front_addr":"172.21.15.70:6807/3986044637","state":["exists","up"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":1904387775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":1904387775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":1904387775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":1904387775}]},"public_addr":"172.21.15.163:6804/1904387775","cluster_addr":"172.21.15.163:6805/1904387775","heartbeat_back_addr":"172.21.15.163:6807/1904387775","heartbeat_front_addr":"172.21.15.163:6806/1904387775","state":["exists","up"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1625318059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":1625318059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":1625318059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1625318059}]},"public_addr":"172.21.15.163:6808/1625318059","cluster_addr":"172.21.15.163:6809/1625318059","heartbeat_back_addr":"172.21.15.163:6811/1625318059","heartbeat_front_addr":"172.21.15.163:6810/1625318059","state":["exists","up"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":2999981402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":2999981402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":2999981402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":2999981402}]},"public_addr":"172.21.15.163:6812/2999981402","cluster_addr":"172.21.15.163:6813/2999981402","heartbeat_back_addr":"172.21.15.163:6815/2999981402","heartbeat_front_addr":"172.21.15.163:6814/2999981402","state":["exists","up"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3710554613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3710554613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3710554613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3710554613}]},"public_addr":"172.21.15.163:6800/3710554613","cluster_addr":"172.21.15.163:6801/3710554613","heartbeat_back_addr":"172.21.15.163:6803/3710554613","heartbeat_front_addr":"172.21.15.163:6802/3710554613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.496124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.974891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.248106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.668425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.919888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.102132+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.952245+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.672498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 10 librados: watch_flush enter 2024-09-16T03:49:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 10 librados: watch_flush exit 2024-09-16T03:49:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 20 client.4244.objecter shutdown clearing up homeless session... 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 10 client.4244.objecter successfully canceled tick 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 msgr2=0x7f13b80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 0x7f13b80700c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f13c800a700 tx=0x7f13c8008040 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 10 monclient: shutdown 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 msgr2=0x7f13d40b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d00046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 shutdown_connections 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> v2:172.21.15.70:6804/198622594 conn(0x7f13b806dc00 0x7f13b80700c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d413cce0 0x7f13d40b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f13d413c900 0x7f13d4137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 --2- 172.21.15.70:0/4140988742 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f13d40ade90 0x7f13d40b2740 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 >> 172.21.15.70:0/4140988742 conn(0x7f13d40b1830 msgr2=0x7f13d40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 shutdown_connections 2024-09-16T03:49:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.533+0000 7f13dbd4b640 1 -- 172.21.15.70:0/4140988742 wait complete. 2024-09-16T03:49:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.534+0000 7f13dbd4b640 1 librados: shutdown 2024-09-16T03:49:56.547 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:49:56.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:49:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.641+0000 7f8287477640 10 monclient: get_monmap_and_config 2024-09-16T03:49:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.641+0000 7f8287477640 10 monclient: build_initial_monmap 2024-09-16T03:49:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.641+0000 7f8287477640 10 monclient: monmap: 2024-09-16T03:49:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:56.644 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:56.643164+0000 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:56.643164+0000 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.642+0000 7f8287477640 1 Processor -- start 2024-09-16T03:49:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.642+0000 7f8287477640 1 -- start start 2024-09-16T03:49:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.642+0000 7f8287477640 10 monclient: init 2024-09-16T03:49:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.642+0000 7f8287477640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.642+0000 7f8287477640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8280067c30 0x7f8280142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): picked mon.noname-a con 0x7f8280067c30 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): picked mon.noname-b con 0x7f82800afdd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82800ae8f0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8280068830 con 0x7f82800ae8f0 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8280068010 con 0x7f8280067c30 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f82800b09f0 con 0x7f82800afdd0 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8287477640 10 monclient(hunting): authenticate will time out at 755.305107s 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8280067c30 0x7f8280142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8280067c30 0x7f8280142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48988/0 (socket says 172.21.15.70:48988) 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 -- 172.21.15.70:0/3200081309 learned_addr learned my addr 172.21.15.70:0/3200081309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): get_auth_request con 0x7f82800afdd0 auth_method 0 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): get_auth_request con 0x7f8280067c30 auth_method 0 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 -- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 msgr2=0x7f8280143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 -- 172.21.15.70:0/3200081309 >> v1:172.21.15.163:6789/0 conn(0x7f82800ae8f0 legacy=0x7f82800aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient: _finish_hunting 0 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 monclient: found mon.noname-a 2024-09-16T03:49:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8285213640 1 -- 172.21.15.70:0/3200081309 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8280073aa0 con 0x7f8280067c30 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.643+0000 7f8284a12640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8285213640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8280067c30 0x7f8280142ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f827c009870 tx=0x7f827c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5b22a46b148b8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 1 -- 172.21.15.70:0/3200081309 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c037070 con 0x7f8280067c30 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: dump: 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 1 -- 172.21.15.70:0/3200081309 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827c032440 con 0x7f8280067c30 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 5 monclient: authenticate success, global_id 4290 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 1 -- 172.21.15.70:0/3200081309 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c032810 con 0x7f8280067c30 2024-09-16T03:49:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 4 set_mon_vals no callback set 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8277fff640 10 monclient: dump: 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 10 monclient: shutdown 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 -- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8280067c30 msgr2=0x7f8280142ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8280067c30 0x7f8280142ef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f827c009870 tx=0x7f827c02a200 comp rx=0 tx=0).stop 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 -- 172.21.15.70:0/3200081309 shutdown_connections 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 --2- 172.21.15.70:0/3200081309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8280067c30 0x7f8280142ef0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 -- 172.21.15.70:0/3200081309 >> 172.21.15.70:0/3200081309 conn(0x7f82800b5e30 msgr2=0x7f82800b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.644+0000 7f8287477640 1 -- 172.21.15.70:0/3200081309 shutdown_connections 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 1 -- 172.21.15.70:0/3200081309 wait complete. 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 10 monclient: build_initial_monmap 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 10 monclient: monmap: 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 1 librados: starting msgr at 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 1 librados: starting objecter 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.645+0000 7f8287477640 1 Processor -- start 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 -- start start 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 librados: setting wanted keys 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 librados: calling monclient init 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient: init 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 0x7f82801cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): picked mon.noname-b con 0x7f8280067c30 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): picked mon.noname-a con 0x7f82800ae8f0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): picked mon.noname-c con 0x7f82800afdd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82800af490 con 0x7f82800ae8f0 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f82800b1090 con 0x7f8280067c30 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f82800af610 con 0x7f82800afdd0 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8287477640 10 monclient(hunting): authenticate will time out at 755.308608s 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48992/0 (socket says 172.21.15.70:48992) 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 -- 172.21.15.70:0/4254584858 learned_addr learned my addr 172.21.15.70:0/4254584858 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): get_auth_request con 0x7f82800ae8f0 auth_method 0 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): get_auth_request con 0x7f82800afdd0 auth_method 0 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285213640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 0x7f82801cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 -- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 msgr2=0x7f8280141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 -- 172.21.15.70:0/4254584858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 msgr2=0x7f82801cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 0x7f82801cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient: _finish_hunting 0 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 monclient: found mon.noname-a 2024-09-16T03:49:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8284a12640 1 -- 172.21.15.70:0/4254584858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82801cf340 con 0x7f82800ae8f0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285213640 10 monclient: get_auth_request con 0x7f8280067c30 auth_method 0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285213640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 0x7f82801cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.646+0000 7f8285a14640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8284a12640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f827000e910 tx=0x7f827000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 client.?.objecter ms_handle_connect 0x7f82800ae8f0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270017070 con 0x7f82800ae8f0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: dump: 2024-09-16T03:49:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827000c9a0 con 0x7f82800ae8f0 2024-09-16T03:49:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 5 monclient: authenticate success, global_id 4293 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 10 monclient: _renew_subs 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82801ccfb0 con 0x7f82800ae8f0 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827000cd50 con 0x7f82800ae8f0 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8275ffb640 10 monclient: dump: 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 10 monclient: _renew_subs 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f82777fe640 4 set_mon_vals no callback set 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82801421c0 con 0x7f82800ae8f0 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.647+0000 7f8287477640 1 librados: init done 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.648+0000 7f8287477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.648+0000 7f8287477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.648+0000 7f8287477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.648+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82800aed80 con 0x7f82800ae8f0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.648+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f827001a070 con 0x7f82800ae8f0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 1 --2- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 0x7f82640700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8285213640 1 --2- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 0x7f82640700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f827008ab00 con 0x7f82800ae8f0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 10 client.4293.objecter ms_dispatch 0x7f8280000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8285213640 10 monclient: get_auth_request con 0x7f826406dc00 auth_method 0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 3 client.4293.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 3 client.4293.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8275ffb640 10 client.4293.objecter ms_handle_connect 0x7f826406dc00 2024-09-16T03:49:56.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.649+0000 7f8285213640 1 --2- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 0x7f82640700c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f827c034000 tx=0x7f827c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:56.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.653+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f82801ccfb0 con 0x7f82800ae8f0 2024-09-16T03:49:56.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.653+0000 7f8275ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:56.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.653+0000 7f8275ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:56.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.821+0000 7f8287477640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.821+0000 7f8287477640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.821+0000 7f8287477640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:56.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.821+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82800b1e50 con 0x7f82800ae8f0 2024-09-16T03:49:56.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.822+0000 7f8275ffb640 1 -- 172.21.15.70:0/4254584858 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+12935 (secure 0 0 0) 0x7f82700592c0 con 0x7f82800ae8f0 2024-09-16T03:49:56.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.822+0000 7f8275ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:49:56.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.822+0000 7f8275ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:56.825 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:49:56.826 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":21,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T03:49:55.850087+0000","last_up_change":"2024-09-16T03:49:46.819202+0000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:49:46.663530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:49:52.440963+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1243681362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1243681362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":1243681362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1243681362}]},"public_addr":"172.21.15.70:6800/1243681362","cluster_addr":"172.21.15.70:6801/1243681362","heartbeat_back_addr":"172.21.15.70:6803/1243681362","heartbeat_front_addr":"172.21.15.70:6802/1243681362","state":["exists","up"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":1499120614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":1499120614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1499120614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":1499120614}]},"public_addr":"172.21.15.70:6813/1499120614","cluster_addr":"172.21.15.70:6814/1499120614","heartbeat_back_addr":"172.21.15.70:6816/1499120614","heartbeat_front_addr":"172.21.15.70:6815/1499120614","state":["exists","up"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":2584301924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2584301924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2584301924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":2584301924}]},"public_addr":"172.21.15.70:6809/2584301924","cluster_addr":"172.21.15.70:6810/2584301924","heartbeat_back_addr":"172.21.15.70:6812/2584301924","heartbeat_front_addr":"172.21.15.70:6811/2584301924","state":["exists","up"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3986044637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3986044637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3986044637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3986044637}]},"public_addr":"172.21.15.70:6805/3986044637","cluster_addr":"172.21.15.70:6806/3986044637","heartbeat_back_addr":"172.21.15.70:6808/3986044637","heartbeat_front_addr":"172.21.15.70:6807/3986044637","state":["exists","up"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":1904387775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":1904387775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":1904387775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":1904387775}]},"public_addr":"172.21.15.163:6804/1904387775","cluster_addr":"172.21.15.163:6805/1904387775","heartbeat_back_addr":"172.21.15.163:6807/1904387775","heartbeat_front_addr":"172.21.15.163:6806/1904387775","state":["exists","up"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1625318059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":1625318059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":1625318059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1625318059}]},"public_addr":"172.21.15.163:6808/1625318059","cluster_addr":"172.21.15.163:6809/1625318059","heartbeat_back_addr":"172.21.15.163:6811/1625318059","heartbeat_front_addr":"172.21.15.163:6810/1625318059","state":["exists","up"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":2999981402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":2999981402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":2999981402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":2999981402}]},"public_addr":"172.21.15.163:6812/2999981402","cluster_addr":"172.21.15.163:6813/2999981402","heartbeat_back_addr":"172.21.15.163:6815/2999981402","heartbeat_front_addr":"172.21.15.163:6814/2999981402","state":["exists","up"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3710554613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3710554613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3710554613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3710554613}]},"public_addr":"172.21.15.163:6800/3710554613","cluster_addr":"172.21.15.163:6801/3710554613","heartbeat_back_addr":"172.21.15.163:6803/3710554613","heartbeat_front_addr":"172.21.15.163:6802/3710554613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.496124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.974891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.248106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.668425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.919888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.102132+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.952245+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.672498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:49:56.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 10 librados: watch_flush enter 2024-09-16T03:49:56.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 10 librados: watch_flush exit 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 20 client.4293.objecter shutdown clearing up homeless session... 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 10 client.4293.objecter successfully canceled tick 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 msgr2=0x7f82640700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 0x7f82640700c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f827c034000 tx=0x7f827c02e000 comp rx=0 tx=0).stop 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 10 monclient: shutdown 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 msgr2=0x7f8280141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f827000e910 tx=0x7f827000edd0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 shutdown_connections 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> v2:172.21.15.70:6804/198622594 conn(0x7f826406dc00 0x7f82640700c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f82800afdd0 0x7f8280141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82800ae8f0 0x7f8280141740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 --2- 172.21.15.70:0/4254584858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8280067c30 0x7f82801cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:56.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 >> 172.21.15.70:0/4254584858 conn(0x7f82800b5e30 msgr2=0x7f8280140300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:56.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 shutdown_connections 2024-09-16T03:49:56.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.824+0000 7f8287477640 1 -- 172.21.15.70:0/4254584858 wait complete. 2024-09-16T03:49:56.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:56.825+0000 7f8287477640 1 librados: shutdown 2024-09-16T03:49:56.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:49:56.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:49:56.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:49:56.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:49:56.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:49:56.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:49:56.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T03:49:56.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7fd30738d640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7fd30738d640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7fd30738d640 10 monclient: monmap: 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.006877+0000 2024-09-16T03:49:57.008 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.006877+0000 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.009 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.004+0000 7fe898d2b640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.004+0000 7fe898d2b640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.004+0000 7fe898d2b640 10 monclient: monmap: 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.006474+0000 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.006474+0000 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.010 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7f3c3759e640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7f3c3759e640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.005+0000 7f3c3759e640 10 monclient: monmap: 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.007476+0000 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.007476+0000 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.012 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.013 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.013 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.013 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.006+0000 7fe898d2b640 1 Processor -- start 2024-09-16T03:49:57.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.006+0000 7fd30738d640 1 Processor -- start 2024-09-16T03:49:57.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.006+0000 7f3c3759e640 1 Processor -- start 2024-09-16T03:49:57.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fe898d2b640 1 -- start start 2024-09-16T03:49:57.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fe898d2b640 10 monclient: init 2024-09-16T03:49:57.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.006+0000 7fd30738d640 1 -- start start 2024-09-16T03:49:57.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7f3c3759e640 1 -- start start 2024-09-16T03:49:57.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7f3c3759e640 10 monclient: init 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.006+0000 7fd30738d640 10 monclient: init 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fd30738d640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fd30738d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2f8006e20 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7f831d382640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7f831d382640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7f831d382640 10 monclient: monmap: 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.008578+0000 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.008578+0000 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.017 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.007+0000 7fd30738d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd2f8034b80 0x7fd2f8030470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8940aa9f0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe8940aaf80 0x7fe8940ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8940aaf80 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8940ad890 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8940abae0 con 0x7fe8940aa9f0 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8940ac970 con 0x7fe8940ad890 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe8940acc50 con 0x7fe8940aaf80 2024-09-16T03:49:57.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe898d2b640 10 monclient(hunting): authenticate will time out at 755.670227s 2024-09-16T03:49:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f831d382640 1 Processor -- start 2024-09-16T03:49:57.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe892ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe8940aaf80 0x7fe8940ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f831d382640 1 -- start start 2024-09-16T03:49:57.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f831d382640 10 monclient: init 2024-09-16T03:49:57.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe893fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe8940ad890 0x7fe8940adcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:35958/0 (socket says 172.21.15.70:35958) 2024-09-16T03:49:57.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe893fff640 1 -- 172.21.15.70:0/506928349 learned_addr learned my addr 172.21.15.70:0/506928349 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T03:49:57.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe892ffd640 10 monclient(hunting): get_auth_request con 0x7fe8940aaf80 auth_method 0 2024-09-16T03:49:57.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe892ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fe892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c30006980 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c30006d60 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f3c30030890 0x7f3c300cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c30030890 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c3002e7c0 con 0x7f3c30006d60 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f3c300c5370 con 0x7f3c30030890 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3c300c2e80 con 0x7f3c30006980 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c3759e640 10 monclient(hunting): authenticate will time out at 755.670784s 2024-09-16T03:49:57.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7f3c36d9d640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f3c30030890 0x7f3c300cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45626/0 (socket says 172.21.15.70:45626) 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 -- 172.21.15.70:0/752533850 learned_addr learned my addr 172.21.15.70:0/752533850 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): get_auth_request con 0x7f3c30006980 auth_method 0 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889650476 0 0) 0x7fe8940ac970 con 0x7fe8940ad890 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): my global_id is 4270 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fbb9f4f8640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fbb9f4f8640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fbb9f4f8640 10 monclient: monmap: 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.010886+0000 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.010886+0000 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.029 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.030 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.030 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.030 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.030 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87c003520 con 0x7fe8940ad890 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518316852 0 0) 0x7fe8940abae0 con 0x7fe8940aa9f0 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 10 monclient(hunting): my global_id is 4299 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8940ac970 con 0x7fe8940aa9f0 2024-09-16T03:49:57.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-16T03:49:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T03:49:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 1 -- 172.21.15.70:0/506928349 >> v1:172.21.15.163:6789/0 conn(0x7fe8940ad890 legacy=0x7fe8940adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f471359c640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f471359c640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f471359c640 10 monclient: monmap: 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.010927+0000 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.010927+0000 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.033 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f83180aa9f0 0x7f83180af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): picked mon.noname-c con 0x7f83180aa9f0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83180af940 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83180aaf80 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8318143660 con 0x7f83180aaf80 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83180abae0 con 0x7f83180af940 2024-09-16T03:49:57.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f83180ac970 con 0x7f83180aa9f0 2024-09-16T03:49:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831d382640 10 monclient(hunting): authenticate will time out at 755.671511s 2024-09-16T03:49:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 1 -- 172.21.15.70:0/506928349 >> v1:172.21.15.70:6789/0 conn(0x7fe8940aa9f0 legacy=0x7fe8940af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c36d9d640 10 monclient(hunting): get_auth_request con 0x7f3c30030890 auth_method 0 2024-09-16T03:49:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3137987286 0 0) 0x7fe8940ac970 con 0x7fe8940aa9f0 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 -- 172.21.15.70:0/752533850 >> v2:172.21.15.163:3300/0 conn(0x7f3c30030890 msgr2=0x7f3c300cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 --2- 172.21.15.70:0/752533850 >> v2:172.21.15.163:3300/0 conn(0x7f3c30030890 0x7f3c300cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 -- 172.21.15.70:0/752533850 >> v1:172.21.15.163:6789/0 conn(0x7f3c30006d60 legacy=0x7f3c3002ee20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f3c3659c640 1 -- 172.21.15.70:0/752533850 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3002c960 con 0x7f3c30006980 2024-09-16T03:49:57.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f831cb81640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f83180aaf80 0x7f8318142f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33682/0 (socket says 172.21.15.70:33682) 2024-09-16T03:49:57.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fe892ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fe892ffd640 1 -- 172.21.15.70:0/506928349 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8940acf30 con 0x7fe8940aaf80 2024-09-16T03:49:57.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f83180aa9f0 0x7f83180af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f561f783640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f83180aa9f0 0x7f83180af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34676/0 (socket says 172.21.15.70:34676) 2024-09-16T03:49:57.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f831cb81640 1 -- 172.21.15.70:0/3821249310 learned_addr learned my addr 172.21.15.70:0/3821249310 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7f561f783640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 10 monclient(hunting): get_auth_request con 0x7f83180aa9f0 auth_method 0 2024-09-16T03:49:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f471359c640 1 Processor -- start 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2f8034b80 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 0x7fd2f80332d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2f80caa10 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f80cae60 con 0x7fd2f8006e20 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f8007200 con 0x7fd2f8034b80 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f80c50e0 con 0x7fd2f80caa10 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.008+0000 7fd30738d640 10 monclient(hunting): authenticate will time out at 755.670930s 2024-09-16T03:49:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.009+0000 7fd305b8a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd2f8034b80 0x7fd2f8030470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd30638b640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd2f8006e20 0x7fd2f8035c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:35954/0 (socket says 172.21.15.70:35954) 2024-09-16T03:49:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd30638b640 1 -- 172.21.15.70:0/215734536 learned_addr learned my addr 172.21.15.70:0/215734536 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fe8927fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f561f783640 10 monclient: monmap: 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.011495+0000 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.011495+0000 2024-09-16T03:49:57.044 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.045 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368417420 0 0) 0x7fd2f80cae60 con 0x7fd2f8006e20 2024-09-16T03:49:57.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792788600 0 0) 0x7f8318143660 con 0x7f83180aaf80 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 10 monclient(hunting): my global_id is 4302 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8300003520 con 0x7f83180aaf80 2024-09-16T03:49:57.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 10 monclient(hunting): my global_id is 4279 2024-09-16T03:49:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2ec003520 con 0x7fd2f8006e20 2024-09-16T03:49:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f471359c640 1 -- start start 2024-09-16T03:49:57.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f471359c640 10 monclient: init 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): get_auth_request con 0x7fd2f8034b80 auth_method 0 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd306b8c640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 0x7fd2f80332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd306b8c640 10 monclient(hunting): get_auth_request con 0x7fd2f80caa10 auth_method 0 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd306b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd306b8c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd306b8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7f561f783640 1 Processor -- start 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.010+0000 7fd305b8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd306b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd306b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd306b8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 1 -- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 msgr2=0x7fd2f80332d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 911907247 0 0) 0x7fd2ec003520 con 0x7fd2f8006e20 2024-09-16T03:49:57.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8317fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8317fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8317fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fbb9f4f8640 1 Processor -- start 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3233131023 0 0) 0x7f8300003520 con 0x7f83180aaf80 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 >> v2:172.21.15.163:3300/0 conn(0x7f83180aa9f0 msgr2=0x7f83180af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 --2- 172.21.15.70:0/3821249310 >> v2:172.21.15.163:3300/0 conn(0x7f83180aa9f0 0x7f83180af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 >> v1:172.21.15.163:6789/0 conn(0x7f83180af940 legacy=0x7f8318142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83180acc50 con 0x7f83180aaf80 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f831d382640 5 monclient: authenticate success, global_id 4302 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f831d382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f83100019d0 con 0x7f83180aaf80 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: dump: 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.054 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 >> v1:172.21.15.70:6789/0 conn(0x7f83180aaf80 legacy=0x7f8318142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe892ffd640 1 --2- 172.21.15.70:0/506928349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8940ad350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8840097a0 tx=0x7fe88402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe47b0242fdf6f04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fbb9f4f8640 1 -- start start 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fbb9f4f8640 10 monclient: init 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8300004bf0 0x7f83000037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): picked mon.b con 0x7f8300004bf0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 0x7f8300004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): picked mon.c con 0x7f8300003dc0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8300005390 0x7f8300005810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): picked mon.a con 0x7f8300005390 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83180ac970 con 0x7f8300005390 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83000060f0 con 0x7f8300004bf0 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f83000062f0 con 0x7f8300003dc0 2024-09-16T03:49:57.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f8316ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f831d382640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884037070 con 0x7fe8940aaf80 2024-09-16T03:49:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe8927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe8927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe8927fc640 10 monclient: dump: 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fe8927fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T03:49:57.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f470c14aa20 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 0x7f470c14b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): picked mon.noname-b con 0x7f470c0aa6e0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f470c1444f0 0x7f470c1449a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): picked mon.noname-c con 0x7f470c1444f0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0ab7a0 con 0x7f470c14aa20 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f470c14ae00 con 0x7f470c1444f0 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0afc60 con 0x7f470c0aa6e0 2024-09-16T03:49:57.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f471359c640 10 monclient(hunting): authenticate will time out at 755.673748s 2024-09-16T03:49:57.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 0x7f470c14b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 0x7f470c14b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45646/0 (socket says 172.21.15.70:45646) 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 1 -- 172.21.15.70:0/3093781112 learned_addr learned my addr 172.21.15.70:0/3093781112 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 10 monclient(hunting): get_auth_request con 0x7f470c0aa6e0 auth_method 0 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f4710b37640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88402abf0 con 0x7fe8940aaf80 2024-09-16T03:49:57.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8300005390 0x7f8300005810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 1 -- 172.21.15.70:0/506928349 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0018a0 con 0x7fe8940aaf80 2024-09-16T03:49:57.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 --2- 172.21.15.70:0/3093781112 >> v2:172.21.15.163:3300/0 conn(0x7f470c1444f0 0x7f470c1449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): get_auth_request con 0x7f8300005390 auth_method 0 2024-09-16T03:49:57.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f561f783640 1 -- start start 2024-09-16T03:49:57.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7f561f783640 10 monclient: init 2024-09-16T03:49:57.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe8927fc640 10 monclient: dump: 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.067 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.068 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.068 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:57.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): get_auth_request con 0x7f470c1444f0 auth_method 0 2024-09-16T03:49:57.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f561813c930 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 0x7f56180ab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): picked mon.noname-a con 0x7f561813cd10 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 0x7f8300004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): get_auth_request con 0x7f8300003dc0 auth_method 0 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f56180abbd0 0x7f56180ade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe898d2b640 5 monclient: authenticate success, global_id 4296 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe898d2b640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fe898d2b640 4 set_mon_vals no callback set 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb980757e0 0x7fbb98140160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbb980757e0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fbb980b0d20 0x7fbb981406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbb980b0d20 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb981464c0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb980b1990 con 0x7fbb981464c0 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb98075bc0 con 0x7fbb980757e0 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9806f3f0 con 0x7fbb980b0d20 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fbb9f4f8640 10 monclient(hunting): authenticate will time out at 755.674788s 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): picked mon.noname-c con 0x7f56180abbd0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56180ac020 con 0x7f561813c930 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f56180afc60 con 0x7f561813cd10 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f561813f2f0 con 0x7f56180abbd0 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561f783640 10 monclient(hunting): authenticate will time out at 755.674538s 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 -- 172.21.15.70:0/3093781112 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 msgr2=0x7f470c14b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 --2- 172.21.15.70:0/3093781112 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 0x7f470c14b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 -- 172.21.15.70:0/3093781112 >> v1:172.21.15.163:6789/0 conn(0x7f470c14aa20 legacy=0x7f470c0af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f4711b39640 1 -- 172.21.15.70:0/3093781112 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c0ab080 con 0x7f470c1444f0 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient(hunting): handle_auth_done global_id 4302 payload 293 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 -- 172.21.15.70:0/3821249310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8300004bf0 msgr2=0x7f83000037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8300004bf0 0x7f83000037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 -- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8300005390 msgr2=0x7f8300005810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8300005390 0x7f8300005810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 monclient: found mon.c 2024-09-16T03:49:57.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 1 -- 172.21.15.70:0/3821249310 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83180acc50 con 0x7f8300003dc0 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f83177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f831cb81640 1 -- 172.21.15.70:0/3821249310 reap_dead start 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9da95640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbb981464c0 0x7fbb98146950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:35988/0 (socket says 172.21.15.70:35988) 2024-09-16T03:49:57.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9da95640 1 -- 172.21.15.70:0/927756877 learned_addr learned my addr 172.21.15.70:0/927756877 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 0x7fd2f80332d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 1 -- 172.21.15.70:0/215734536 >> v1:172.21.15.163:6789/0 conn(0x7fd2f8006e20 legacy=0x7fd2f8035c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.011+0000 7fd305b8a640 1 -- 172.21.15.70:0/215734536 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f8035730 con 0x7fd2f8034b80 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fd305389640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fd306b8c640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 0x7fd2f80332d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:57.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7fd305b8a640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f8034b80 0x7fd2f8030470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00097a0 tx=0x7fd2f002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125d232af7eb7feb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 10 monclient: shutdown 2024-09-16T03:49:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 -- 172.21.15.70:0/506928349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 msgr2=0x7fe8940ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 --2- 172.21.15.70:0/506928349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8940ad350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8840097a0 tx=0x7fe88402a160 comp rx=0 tx=0).stop 2024-09-16T03:49:57.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/927756877 >> v2:172.21.15.163:3300/0 conn(0x7fbb980b0d20 0x7fbb981406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): get_auth_request con 0x7fbb980b0d20 auth_method 0 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f56180abbd0 0x7f56180ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f56180abbd0 0x7f56180ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34700/0 (socket says 172.21.15.70:34700) 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0037070 con 0x7fd2f8034b80 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: dump: 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.081 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f002abc0 con 0x7fd2f8034b80 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 1 -- 172.21.15.70:0/768171909 learned_addr learned my addr 172.21.15.70:0/768171909 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 10 monclient(hunting): get_auth_request con 0x7f56180abbd0 auth_method 0 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7f561dd20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7efdf3353640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7efdf3353640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.012+0000 7efdf3353640 10 monclient: monmap: 2024-09-16T03:49:57.083 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.013931+0000 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.013931+0000 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.084 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 1 -- 172.21.15.70:0/215734536 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc001670 con 0x7fd2f8034b80 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd305389640 10 monclient: dump: 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.085 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd30738d640 5 monclient: authenticate success, global_id 4305 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd30738d640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fd30738d640 4 set_mon_vals no callback set 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561cd1e640 1 --2- 172.21.15.70:0/768171909 >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 0x7f56180ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 1 -- 172.21.15.70:0/768171909 >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 msgr2=0x7f56180ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 -- 172.21.15.70:0/506928349 shutdown_connections 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 --2- 172.21.15.70:0/506928349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8940ad350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 -- 172.21.15.70:0/506928349 >> 172.21.15.70:0/506928349 conn(0x7fe894068050 msgr2=0x7fe8940b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7efdf3353640 1 Processor -- start 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522524003 0 0) 0x7fbb980b1990 con 0x7fbb981464c0 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient(hunting): my global_id is 4291 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb84003520 con 0x7fbb981464c0 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 1 --2- 172.21.15.70:0/768171909 >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 0x7f56180ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 1 -- 172.21.15.70:0/768171909 >> v1:172.21.15.70:6789/0 conn(0x7f561813c930 legacy=0x7f56180af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 -- 172.21.15.70:0/506928349 shutdown_connections 2024-09-16T03:49:57.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2277658705 0 0) 0x7fbb84003520 con 0x7fbb981464c0 2024-09-16T03:49:57.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 >> v2:172.21.15.163:3300/0 conn(0x7fbb980b0d20 msgr2=0x7fbb981406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 --2- 172.21.15.70:0/927756877 >> v2:172.21.15.163:3300/0 conn(0x7fbb980b0d20 0x7fbb981406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f3c3659c640 1 --2- 172.21.15.70:0/752533850 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009870 tx=0x7f3c2802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4547af74b582a2d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 >> v2:172.21.15.70:3300/0 conn(0x7fbb980757e0 msgr2=0x7fbb98140160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 --2- 172.21.15.70:0/927756877 >> v2:172.21.15.70:3300/0 conn(0x7fbb980757e0 0x7fbb98140160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.163:6789/0 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb980b0a90 con 0x7fbb981464c0 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9f4f8640 5 monclient: authenticate success, global_id 4291 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb9f4f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7fbb88001650 con 0x7fbb981464c0 2024-09-16T03:49:57.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561dd20640 1 -- 172.21.15.70:0/768171909 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56180ad470 con 0x7f56180abbd0 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fbb8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient: dump: 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.093 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] but i'm connected to v1:172.21.15.163:6789/0, reconnecting 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 >> v1:172.21.15.163:6789/0 conn(0x7fbb981464c0 legacy=0x7fbb98146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84003a10 0x7fbb84003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): picked mon.b con 0x7fbb84003a10 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 0x7fbb84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): picked mon.a con 0x7fbb84004800 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb840051c0 0x7fbb8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): picked mon.c con 0x7fbb840051c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb98075bc0 con 0x7fbb84004800 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8400c180 con 0x7fbb84003a10 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbb84014b90 con 0x7fbb840051c0 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb8ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:57.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9f4f8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561cd1e640 10 monclient: get_auth_request con 0x7f561813cd10 auth_method 0 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 1 -- 172.21.15.70:0/752533850 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c28037070 con 0x7f3c30006980 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: dump: 2024-09-16T03:49:57.096 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3759e640 5 monclient: authenticate success, global_id 4253 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 1 -- 172.21.15.70:0/752533850 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c28032440 con 0x7f3c30006980 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 1 -- 172.21.15.70:0/752533850 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c28037070 con 0x7f3c30006980 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3559a640 10 monclient: dump: 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.098 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3759e640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f3c3759e640 4 set_mon_vals no callback set 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 10 monclient: shutdown 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 -- 172.21.15.70:0/752533850 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 msgr2=0x7f3c300c2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 --2- 172.21.15.70:0/752533850 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3c28009870 tx=0x7f3c2802a180 comp rx=0 tx=0).stop 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7fe898d2b640 1 -- 172.21.15.70:0/506928349 wait complete. 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fe898d2b640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fe898d2b640 10 monclient: monmap: 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.100 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 librados: starting msgr at 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 librados: starting objecter 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 Processor -- start 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 0x7fbb84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): get_auth_request con 0x7fbb84004800 auth_method 0 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84003a10 0x7fbb84003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): get_auth_request con 0x7fbb84003a10 auth_method 0 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient(hunting): handle_auth_done global_id 4291 payload 293 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 -- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb840051c0 msgr2=0x7fbb8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb840051c0 0x7fbb8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 -- 172.21.15.70:0/927756877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84003a10 msgr2=0x7fbb84003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84003a10 0x7fbb84003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 monclient: found mon.a 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 1 -- 172.21.15.70:0/927756877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb84014e50 con 0x7fbb84004800 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9ca93640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fbb9da95640 1 -- 172.21.15.70:0/927756877 reap_dead start 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4711b39640 1 --2- 172.21.15.70:0/3093781112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c1444f0 0x7f470c1449a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f470c0aaff0 tx=0x7f470802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=828679088ad2b4a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 1 -- 172.21.15.70:0/3093781112 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708037070 con 0x7f470c1444f0 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: dump: 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.104 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 1 -- 172.21.15.70:0/3093781112 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470802ae20 con 0x7f470c1444f0 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 1 -- 172.21.15.70:0/3093781112 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708032660 con 0x7f470c1444f0 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f4703fff640 10 monclient: dump: 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f471359c640 5 monclient: authenticate success, global_id 4285 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f471359c640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f471359c640 4 set_mon_vals no callback set 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fd30738d640 10 monclient: shutdown 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fd30738d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fd30738d640 1 -- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f8034b80 msgr2=0x7fd2f8030470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7fd30738d640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f8034b80 0x7fd2f8030470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00097a0 tx=0x7fd2f002a130 comp rx=0 tx=0).stop 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7f561cd1e640 1 --2- 172.21.15.70:0/768171909 >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 0x7f56180ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7f561dd20640 1 --2- 172.21.15.70:0/768171909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56180abbd0 0x7f56180ade20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56180ab0d0 tx=0x7f561002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca63665b5e1f877 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 1 -- 172.21.15.70:0/768171909 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610036070 con 0x7f56180abbd0 2024-09-16T03:49:57.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: dump: 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.108 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 1 -- 172.21.15.70:0/768171909 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5610031600 con 0x7f56180abbd0 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 1 -- 172.21.15.70:0/768171909 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100319d0 con 0x7f56180abbd0 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f560ffff640 10 monclient: dump: 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.109 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f561f783640 5 monclient: authenticate success, global_id 4288 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f561f783640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f561f783640 4 set_mon_vals no callback set 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7efdf3353640 1 -- start start 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.013+0000 7efdf3353640 10 monclient: init 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 0x7efdec0af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-a con 0x7efdec0aaa40 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdec0af940 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7efdec142f50 0x7efdec149a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-c con 0x7efdec142f50 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdec0aba90 con 0x7efdec0af940 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efdec0aae20 con 0x7efdec0aaa40 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7efdec0736d0 con 0x7efdec142f50 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.014+0000 7efdf3353640 10 monclient(hunting): authenticate will time out at 755.676851s 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7efdec142f50 0x7efdec149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7efdec142f50 0x7efdec149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34718/0 (socket says 172.21.15.70:34718) 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 -- 172.21.15.70:0/1432015301 learned_addr learned my addr 172.21.15.70:0/1432015301 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): get_auth_request con 0x7efdec142f50 auth_method 0 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf10ef640 1 --2- 172.21.15.70:0/1432015301 >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 0x7efdec0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf10ef640 10 monclient(hunting): get_auth_request con 0x7efdec0aaa40 auth_method 0 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 1 -- 172.21.15.70:0/1432015301 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061647892 0 0) 0x7efdec0aba90 con 0x7efdec0af940 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf10ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf10ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf10ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 10 monclient(hunting): my global_id is 4300 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efddffff640 1 -- 172.21.15.70:0/1432015301 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdd8004630 con 0x7efdec0af940 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 -- 172.21.15.70:0/1432015301 >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 msgr2=0x7efdec0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 0x7fbb84004c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009fa0 tx=0x7fbb84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a797ae148fef8886 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 --2- 172.21.15.70:0/1432015301 >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 0x7efdec0af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 -- 172.21.15.70:0/1432015301 >> v1:172.21.15.163:6789/0 conn(0x7efdec0af940 legacy=0x7efdec142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7efdf18f0640 1 -- 172.21.15.70:0/1432015301 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec0ac9b0 con 0x7efdec142f50 2024-09-16T03:49:57.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f83177fe640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 0x7f8300004240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f830c001700 tx=0x7f8300004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ddf138669b7f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 -- 172.21.15.70:0/752533850 shutdown_connections 2024-09-16T03:49:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f471359c640 10 monclient: shutdown 2024-09-16T03:49:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f471359c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f471359c640 1 -- 172.21.15.70:0/3093781112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c1444f0 msgr2=0x7f470c1449a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f471359c640 1 --2- 172.21.15.70:0/3093781112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c1444f0 0x7f470c1449a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f470c0aaff0 tx=0x7f470802a110 comp rx=0 tx=0).stop 2024-09-16T03:49:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 --2- 172.21.15.70:0/752533850 >> v2:172.21.15.163:3300/0 conn(0x7f3c30030890 0x7f3c300cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 --2- 172.21.15.70:0/752533850 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30006980 0x7f3c300c2940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7f3c3759e640 1 -- 172.21.15.70:0/752533850 >> 172.21.15.70:0/752533850 conn(0x7f3c30005a40 msgr2=0x7f3c30005e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c001350 con 0x7f8300003dc0 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 10 monclient: dump: 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 1 -- 172.21.15.70:0/3821249310 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830c0052f0 con 0x7f8300003dc0 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f8316ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94009bf0 con 0x7fbb84004800 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: dump: 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb94036070 con 0x7fbb84004800 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 1 -- 172.21.15.70:0/927756877 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94030850 con 0x7fbb84004800 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb8ffff640 10 monclient: dump: 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 4 set_mon_vals no callback set 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 -- 172.21.15.70:0/3093781112 shutdown_connections 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 --2- 172.21.15.70:0/3093781112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c1444f0 0x7f470c1449a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 --2- 172.21.15.70:0/3093781112 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c0aa6e0 0x7f470c14b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 -- 172.21.15.70:0/3093781112 >> 172.21.15.70:0/3093781112 conn(0x7f470c0b1830 msgr2=0x7f470c0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 -- 172.21.15.70:0/3093781112 shutdown_connections 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 10 monclient: shutdown 2024-09-16T03:49:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7efdf10ef640 1 --2- 172.21.15.70:0/1432015301 >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 0x7efdec0af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:57.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 -- 172.21.15.70:0/768171909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56180abbd0 msgr2=0x7f56180ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 --2- 172.21.15.70:0/768171909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56180abbd0 0x7f56180ade20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56180ab0d0 tx=0x7f561002a6f0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 -- 172.21.15.70:0/768171909 shutdown_connections 2024-09-16T03:49:57.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 --2- 172.21.15.70:0/768171909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f56180abbd0 0x7f56180ade20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 --2- 172.21.15.70:0/768171909 >> v2:172.21.15.70:3300/0 conn(0x7f561813cd10 0x7f56180ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 -- 172.21.15.70:0/768171909 >> 172.21.15.70:0/768171909 conn(0x7f56180b1830 msgr2=0x7f56180b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 -- start start 2024-09-16T03:49:57.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 librados: setting wanted keys 2024-09-16T03:49:57.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 1 librados: calling monclient init 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.015+0000 7fe898d2b640 10 monclient: init 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8940aa9f0 0x7fe89413e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8940aa9f0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8941e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8940aaf80 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8940ad890 0x7fe8941e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8940ad890 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe894068500 con 0x7fe8940aaf80 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8940afb10 con 0x7fe8940aa9f0 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fe894068680 con 0x7fe8940ad890 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe898d2b640 10 monclient(hunting): authenticate will time out at 755.678564s 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 -- 172.21.15.70:0/215734536 shutdown_connections 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f80caa10 0x7fd2f80332d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 --2- 172.21.15.70:0/215734536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f8034b80 0x7fd2f8030470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 -- 172.21.15.70:0/215734536 >> 172.21.15.70:0/215734536 conn(0x7fd2f8005d50 msgr2=0x7fd2f8006160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 -- 172.21.15.70:0/768171909 shutdown_connections 2024-09-16T03:49:57.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f561f783640 1 -- 172.21.15.70:0/768171909 wait complete. 2024-09-16T03:49:57.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f831d382640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f831d382640 4 set_mon_vals no callback set 2024-09-16T03:49:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 -- 172.21.15.70:0/215734536 shutdown_connections 2024-09-16T03:49:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f3c3759e640 1 -- 172.21.15.70:0/752533850 shutdown_connections 2024-09-16T03:49:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fd30738d640 1 -- 172.21.15.70:0/215734536 wait complete. 2024-09-16T03:49:57.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f3c3759e640 1 -- 172.21.15.70:0/752533850 wait complete. 2024-09-16T03:49:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 10 monclient: shutdown 2024-09-16T03:49:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 msgr2=0x7fbb84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fbb9f4f8640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 0x7fbb84004c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009fa0 tx=0x7fbb84005830 comp rx=0 tx=0).stop 2024-09-16T03:49:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7f471359c640 1 -- 172.21.15.70:0/3093781112 wait complete. 2024-09-16T03:49:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 10 monclient: shutdown 2024-09-16T03:49:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 10 monclient: monmap: 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.131 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 1 -- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 msgr2=0x7f8300004240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 0x7f8300004240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f830c001700 tx=0x7f8300004820 comp rx=0 tx=0).stop 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 librados: starting msgr at 2024-09-16T03:49:57.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 librados: starting objecter 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f561f783640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 1 -- 172.21.15.70:0/3821249310 shutdown_connections 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 1 --2- 172.21.15.70:0/3821249310 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8300003dc0 0x7f8300004240 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f830c001700 tx=0x7f8300004820 comp rx=0 tx=0).stop 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f831d382640 1 -- 172.21.15.70:0/3821249310 >> 172.21.15.70:0/3821249310 conn(0x7f8318068050 msgr2=0x7f83180b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 Processor -- start 2024-09-16T03:49:57.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 -- start start 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 librados: setting wanted keys 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 1 librados: calling monclient init 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f471359c640 10 monclient: init 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fd30738d640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fd30738d640 10 monclient: monmap: 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7f561f783640 10 monclient: monmap: 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7efdf18f0640 1 --2- 172.21.15.70:0/1432015301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec142f50 0x7efdec149a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdec0aa6b0 tx=0x7efde002a740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9adba63b29c2a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f831d382640 1 -- 172.21.15.70:0/3821249310 shutdown_connections 2024-09-16T03:49:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 1 -- 172.21.15.70:0/1432015301 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0036070 con 0x7efdec142f50 2024-09-16T03:49:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f3c3759e640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f3c3759e640 10 monclient: monmap: 2024-09-16T03:49:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.138 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: dump: 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.016+0000 7fe893fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8940ad890 0x7fe8941e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8941e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8940aa9f0 0x7fe89413e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 librados: starting msgr at 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8941e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49064/0 (socket says 172.21.15.70:49064) 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 1 -- 172.21.15.70:0/4219226857 learned_addr learned my addr 172.21.15.70:0/4219226857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): get_auth_request con 0x7fe8940aaf80 auth_method 0 2024-09-16T03:49:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): get_auth_request con 0x7fe8940aa9f0 auth_method 0 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/927756877 shutdown_connections 2024-09-16T03:49:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fbb9f4f8640 1 --2- 172.21.15.70:0/927756877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb84004800 0x7fbb84004c80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 librados: starting objecter 2024-09-16T03:49:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe8937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.017+0000 7fe892ffd640 1 -- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8940ad890 msgr2=0x7fe8941e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 --2- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8940ad890 0x7fe8941e8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 -- 172.21.15.70:0/4219226857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8940aa9f0 msgr2=0x7fe89413e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 --2- 172.21.15.70:0/4219226857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8940aa9f0 0x7fe89413e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 -- 172.21.15.70:0/4219226857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89412ecc0 con 0x7fe8940aaf80 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe893fff640 10 monclient: get_auth_request con 0x7fe8940ad890 auth_method 0 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe893fff640 1 --2- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8940ad890 0x7fe8941e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/927756877 >> 172.21.15.70:0/927756877 conn(0x7fbb980b5300 msgr2=0x7fbb980b5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efdf3353640 5 monclient: authenticate success, global_id 4297 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efdf3353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 1 -- 172.21.15.70:0/1432015301 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde0031600 con 0x7efdec142f50 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efddffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 Processor -- start 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 -- start start 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 librados: starting msgr at 2024-09-16T03:49:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 librados: starting objecter 2024-09-16T03:49:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 Processor -- start 2024-09-16T03:49:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 librados: setting wanted keys 2024-09-16T03:49:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/927756877 shutdown_connections 2024-09-16T03:49:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 1 librados: calling monclient init 2024-09-16T03:49:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f561f783640 10 monclient: init 2024-09-16T03:49:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/927756877 wait complete. 2024-09-16T03:49:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efdf3353640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7efdf3353640 4 set_mon_vals no callback set 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 10 monclient: shutdown 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 -- 172.21.15.70:0/1432015301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec142f50 msgr2=0x7efdec149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 --2- 172.21.15.70:0/1432015301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec142f50 0x7efdec149a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efdec0aa6b0 tx=0x7efde002a740 comp rx=0 tx=0).stop 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 -- 172.21.15.70:0/1432015301 shutdown_connections 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 --2- 172.21.15.70:0/1432015301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec142f50 0x7efdec149a90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 --2- 172.21.15.70:0/1432015301 >> v2:172.21.15.70:3300/0 conn(0x7efdec0aaa40 0x7efdec0af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 -- 172.21.15.70:0/1432015301 >> 172.21.15.70:0/1432015301 conn(0x7efdec068020 msgr2=0x7efdec0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 0x7f56180708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): picked mon.noname-a con 0x7f56180abbd0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561813c930 0x7f56181d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): picked mon.noname-b con 0x7f561813c930 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f561813cd10 0x7f56181d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): picked mon.noname-c con 0x7f561813cd10 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56180ac560 con 0x7f56180abbd0 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f56180ac280 con 0x7f561813c930 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f56180ae360 con 0x7f561813cd10 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561f783640 10 monclient(hunting): authenticate will time out at 755.681578s 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 -- start start 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 librados: setting wanted keys 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 1 librados: calling monclient init 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fd30738d640 10 monclient: init 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 0x7fd2f80c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fbb9f4f8640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fbb9f4f8640 10 monclient: monmap: 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.153 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2f8006e20 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f8034b80 0x7fd2f802ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 0x7f56180708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 0x7f56180708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49076/0 (socket says 172.21.15.70:49076) 2024-09-16T03:49:57.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 -- 172.21.15.70:0/1432015301 shutdown_connections 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f3c3759e640 1 librados: starting msgr at 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f3c3759e640 1 librados: starting objecter 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 1 -- 172.21.15.70:0/2770056566 learned_addr learned my addr 172.21.15.70:0/2770056566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): get_auth_request con 0x7f56180abbd0 auth_method 0 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30638b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 0x7fd2f80c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f561d51f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2f8034b80 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f831d382640 1 -- 172.21.15.70:0/3821249310 wait complete. 2024-09-16T03:49:57.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f3c3759e640 1 Processor -- start 2024-09-16T03:49:57.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30638b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 0x7fd2f80c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34762/0 (socket says 172.21.15.70:34762) 2024-09-16T03:49:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fd30738d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f80caa10 0x7fd2f80c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:57.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c0aa6e0 0x7f470c1452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): picked mon.noname-b con 0x7f470c0aa6e0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f470c1444f0 0x7f470c1d7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): picked mon.noname-a con 0x7f470c1444f0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c14aa20 0x7f470c1d8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): picked mon.noname-c con 0x7f470c14aa20 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f470c151b40 con 0x7f470c1444f0 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f470c0aba40 con 0x7f470c0aa6e0 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f470c151cc0 con 0x7f470c14aa20 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f471359c640 10 monclient(hunting): authenticate will time out at 755.680637s 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711b39640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c14aa20 0x7f470c1d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c0aa6e0 0x7f470c1452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c0aa6e0 0x7f470c1452a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34740/0 (socket says 172.21.15.70:34740) 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 1 -- 172.21.15.70:0/762772608 learned_addr learned my addr 172.21.15.70:0/762772608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): get_auth_request con 0x7f470c0aa6e0 auth_method 0 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711b39640 10 monclient(hunting): get_auth_request con 0x7f470c14aa20 auth_method 0 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711b39640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7f4711338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 -- 172.21.15.70:0/762772608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c0aa6e0 msgr2=0x7f470c1452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f470c0aa6e0 0x7f470c1452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 -- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f470c1444f0 msgr2=0x7f470c1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4710b37640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f470c1444f0 0x7f470c1d7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f470c1444f0 0x7f470c1d7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 -- 172.21.15.70:0/3875884545 learned_addr learned my addr 172.21.15.70:0/3875884545 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f3c3759e640 1 -- start start 2024-09-16T03:49:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7f4711b39640 1 -- 172.21.15.70:0/762772608 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c146dd0 con 0x7f470c14aa20 2024-09-16T03:49:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f3c3759e640 1 librados: setting wanted keys 2024-09-16T03:49:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f3c3759e640 1 librados: calling monclient init 2024-09-16T03:49:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f3c3759e640 10 monclient: init 2024-09-16T03:49:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2f80caa10 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f80d1da0 con 0x7fd2f80caa10 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f8033a60 con 0x7fd2f8006e20 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f80d1f20 con 0x7fd2f8034b80 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30738d640 10 monclient(hunting): authenticate will time out at 755.682133s 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): get_auth_request con 0x7fd2f8006e20 auth_method 0 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f4710b37640 10 monclient: get_auth_request con 0x7f470c1444f0 auth_method 0 2024-09-16T03:49:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f4710b37640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f470c1444f0 0x7f470c1d7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 librados: starting msgr at 2024-09-16T03:49:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 librados: starting objecter 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe892ffd640 1 --2- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8941e83e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe88402a750 tx=0x7fe8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 client.?.objecter ms_handle_connect 0x7fe8940aaf80 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884042020 con 0x7fe8940aaf80 2024-09-16T03:49:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: dump: 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f4711b39640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c14aa20 0x7f470c1d8290 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4708031000 tx=0x7f47080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f4701ffb640 10 client.?.objecter ms_handle_connect 0x7f470c14aa20 2024-09-16T03:49:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f4701ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88403d730 con 0x7fe8940aaf80 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe87bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.018+0000 7fe891ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884037040 con 0x7fe8940aaf80 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fe87bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fe87bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7fe87bfff640 10 monclient: dump: 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd306b8c640 1 --2- 172.21.15.70:0/3875884545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f80caa10 0x7fd2f80c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd306b8c640 10 monclient(hunting): get_auth_request con 0x7fd2f80caa10 auth_method 0 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd306b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd306b8c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd306b8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 Processor -- start 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 5 monclient: authenticate success, global_id 4311 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 10 monclient: _renew_subs 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 1 -- 172.21.15.70:0/4219226857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe894070f60 con 0x7fe8940aaf80 2024-09-16T03:49:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 -- start start 2024-09-16T03:49:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 librados: setting wanted keys 2024-09-16T03:49:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 1 librados: calling monclient init 2024-09-16T03:49:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fbb9f4f8640 10 monclient: init 2024-09-16T03:49:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 10 monclient: _renew_subs 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 1 -- 172.21.15.70:0/4219226857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe894071720 con 0x7fe8940aaf80 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fe898d2b640 1 librados: init done 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f831d382640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f831d382640 10 monclient: monmap: 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.175 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561813c930 0x7f56181d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): get_auth_request con 0x7f561813c930 auth_method 0 2024-09-16T03:49:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.019+0000 7efdf3353640 1 -- 172.21.15.70:0/1432015301 wait complete. 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7efdf3353640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7efdf3353640 10 monclient: monmap: 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.177 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 librados: starting msgr at 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 librados: starting objecter 2024-09-16T03:49:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c30006980 0x7f3c300c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 Processor -- start 2024-09-16T03:49:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c30006980 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30030890 0x7f3c300c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c30030890 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 0x7f3c30160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c300c6bb0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3002f6b0 con 0x7f3c30006980 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 librados: starting msgr at 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 librados: starting objecter 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 Processor -- start 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3002f530 con 0x7f3c300c6bb0 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3c3002f830 con 0x7f3c30030890 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c3759e640 10 monclient(hunting): authenticate will time out at 755.683490s 2024-09-16T03:49:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 0x7fbb9813daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f561813cd10 0x7f56181d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561cd1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): get_auth_request con 0x7f561813cd10 auth_method 0 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561dd20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 -- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f561813cd10 msgr2=0x7f56181d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f561813cd10 0x7f56181d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 -- 172.21.15.70:0/2770056566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561813c930 msgr2=0x7f56181d9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f561813c930 0x7f56181d9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7f561d51f640 1 -- 172.21.15.70:0/2770056566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56181d9670 con 0x7f56180abbd0 2024-09-16T03:49:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 0x7f3c30160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbb84004800 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 0x7f3c30160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34768/0 (socket says 172.21.15.70:34768) 2024-09-16T03:49:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb981437a0 0x7fbb98142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 1 -- 172.21.15.70:0/3609293306 learned_addr learned my addr 172.21.15.70:0/3609293306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 10 monclient(hunting): get_auth_request con 0x7f3c300c6bb0 auth_method 0 2024-09-16T03:49:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbb981437a0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9d294640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 0x7fbb9813daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fbb9d294640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 0x7fbb9813daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34770/0 (socket says 172.21.15.70:34770) 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb981d3e30 0x7fbb981d4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 -- start start 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 librados: setting wanted keys 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 1 librados: calling monclient init 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f831d382640 10 monclient: init 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 15 client.4311.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fe898d2b640 1 -- 172.21.15.70:0/4219226857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe8940ad350 con 0x7fe8940aaf80 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbb981d3e30 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 -- 172.21.15.70:0/2595529247 learned_addr learned my addr 172.21.15.70:0/2595529247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb98146ce0 con 0x7fbb981d3e30 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb98146b60 con 0x7fbb84004800 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbb98146e60 con 0x7fbb981437a0 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9f4f8640 10 monclient(hunting): authenticate will time out at 755.684274s 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): get_auth_request con 0x7fbb84004800 auth_method 0 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708042070 con 0x7f470c14aa20 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: dump: 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.194 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470803e680 con 0x7f470c14aa20 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f47037fe640 4 set_mon_vals no callback set 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708037040 con 0x7f470c14aa20 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f4701ffb640 10 monclient: dump: 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2024-09-16T03:49:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 -- 172.21.15.70:0/3875884545 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f8034b80 msgr2=0x7fd2f802ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 --2- 172.21.15.70:0/3875884545 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd2f8034b80 0x7fd2f802ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 -- 172.21.15.70:0/3875884545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f80caa10 msgr2=0x7fd2f80c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 --2- 172.21.15.70:0/3875884545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2f80caa10 0x7fd2f80c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.020+0000 7fd30638b640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f802fa10 con 0x7fd2f8006e20 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd30638b640 1 --2- 172.21.15.70:0/3875884545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 0x7fd2f80c7b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc001380 tx=0x7fd2fc00cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2f8006e20 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc017070 con 0x7fd2f8006e20 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: dump: 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.198 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2fc005af0 con 0x7fd2f8006e20 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd2f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7fd304b88640 4 set_mon_vals no callback set 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc007070 con 0x7fd2f8006e20 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd2f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd2f6ffd640 10 monclient: dump: 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.199 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 5 monclient: authenticate success, global_id 4309 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 10 monclient: _renew_subs 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f80300f0 con 0x7fd2f8006e20 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 10 monclient: _renew_subs 2024-09-16T03:49:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f802f4f0 con 0x7fd2f8006e20 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fd30738d640 1 librados: init done 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 -- start start 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 librados: setting wanted keys 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 1 librados: calling monclient init 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7efdf3353640 10 monclient: init 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efdec0aaa40 0x7efdec0726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-c con 0x7efdec0aaa40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 0x7efdec1e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-b con 0x7efdec0af940 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdec12e070 0x7efdec1e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): picked mon.noname-a con 0x7efdec12e070 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efdec0ab640 con 0x7efdec12e070 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efdec0acf70 con 0x7efdec0af940 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7efdec0ab030 con 0x7efdec0aaa40 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf3353640 10 monclient(hunting): authenticate will time out at 755.684160s 2024-09-16T03:49:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 0x7efdec1e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8318128d60 0x7f8318129140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 10 monclient(hunting): picked mon.noname-c con 0x7f8318128d60 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8318129680 0x7f83181e08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f831d382640 10 monclient(hunting): picked mon.noname-b con 0x7f8318129680 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83181e0e10 0x7f83181dcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8318128d60 0x7f8318129140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8318128d60 0x7f8318129140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45748/0 (socket says 172.21.15.70:45748) 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 -- 172.21.15.70:0/2595529247 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb981437a0 msgr2=0x7fbb98142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 --2- 172.21.15.70:0/2595529247 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbb981437a0 0x7fbb98142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 -- 172.21.15.70:0/2595529247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb981d3e30 msgr2=0x7fbb981d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 --2- 172.21.15.70:0/2595529247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb981d3e30 0x7fbb981d4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7fbb9d294640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb98142ed0 con 0x7fbb84004800 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 5 monclient: authenticate success, global_id 4262 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 10 monclient: _renew_subs 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 1 -- 172.21.15.70:0/762772608 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1470a0 con 0x7f470c14aa20 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 10 monclient: _renew_subs 2024-09-16T03:49:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f471359c640 1 -- 172.21.15.70:0/762772608 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1d87d0 con 0x7f470c14aa20 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): picked mon.noname-a con 0x7f83181e0e10 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8318068500 con 0x7f83181e0e10 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f83180ab690 con 0x7f8318129680 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f8318068680 con 0x7f8318128d60 2024-09-16T03:49:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f471359c640 1 librados: init done 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831d382640 10 monclient(hunting): authenticate will time out at 755.685137s 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 1 -- 172.21.15.70:0/4122111205 learned_addr learned my addr 172.21.15.70:0/4122111205 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 10 monclient(hunting): get_auth_request con 0x7f8318128d60 auth_method 0 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f8317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 0x7efdec1e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34778/0 (socket says 172.21.15.70:34778) 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 1 -- 172.21.15.70:0/2983094982 learned_addr learned my addr 172.21.15.70:0/2983094982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): get_auth_request con 0x7efdec0af940 auth_method 0 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf10ef640 1 --2- 172.21.15.70:0/2983094982 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efdec0aaa40 0x7efdec0726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf10ef640 10 monclient(hunting): get_auth_request con 0x7efdec0aaa40 auth_method 0 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf10ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf10ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf10ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7efdf08ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf10ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf10ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf10ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 -- 172.21.15.70:0/2983094982 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efdec0aaa40 msgr2=0x7efdec0726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 --2- 172.21.15.70:0/2983094982 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efdec0aaa40 0x7efdec0726b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 -- 172.21.15.70:0/2983094982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdec12e070 msgr2=0x7efdec1e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 --2- 172.21.15.70:0/2983094982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efdec12e070 0x7efdec1e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 -- 172.21.15.70:0/2983094982 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec12f2b0 con 0x7efdec0af940 2024-09-16T03:49:57.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f561d51f640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 0x7f56180708e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f561400c460 tx=0x7f561400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f560dffb640 10 client.?.objecter ms_handle_connect 0x7f56180abbd0 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 client.?.objecter ms_handle_connect 0x7fbb84004800 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb9d294640 1 --2- 172.21.15.70:0/2595529247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 0x7fbb9813daa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000e5f0 tx=0x7fbb9000eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f560dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90017070 con 0x7fbb84004800 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: dump: 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.212 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.021+0000 7f3c36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c3659c640 1 --2- 172.21.15.70:0/3609293306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c30006980 0x7f3c300c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c3659c640 10 monclient(hunting): get_auth_request con 0x7f3c30006980 auth_method 0 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c3659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.022+0000 7f3c3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 -- 172.21.15.70:0/3609293306 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30030890 msgr2=0x7f3c300c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 --2- 172.21.15.70:0/3609293306 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3c30030890 0x7f3c300c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 -- 172.21.15.70:0/3609293306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c30006980 msgr2=0x7f3c300c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 --2- 172.21.15.70:0/3609293306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c30006980 0x7f3c300c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 -- 172.21.15.70:0/3609293306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c300c8790 con 0x7f3c300c6bb0 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb90013680 con 0x7fbb84004800 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb9f4f8640 5 monclient: authenticate success, global_id 4318 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90017070 con 0x7fbb84004800 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8dffb640 10 monclient: dump: 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb9f4f8640 10 monclient: _renew_subs 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb9f4f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb98143be0 con 0x7fbb84004800 2024-09-16T03:49:57.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fbb8f7fe640 4 set_mon_vals no callback set 2024-09-16T03:49:57.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83181e0e10 0x7f83181dcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): get_auth_request con 0x7f83181e0e10 auth_method 0 2024-09-16T03:49:57.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8318129680 0x7f83181e08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): get_auth_request con 0x7f8318129680 auth_method 0 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f83177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 -- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8318128d60 msgr2=0x7f8318129140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8318128d60 0x7f8318129140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 -- 172.21.15.70:0/4122111205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8318129680 msgr2=0x7f83181e08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8318129680 0x7f83181e08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f831cb81640 1 -- 172.21.15.70:0/4122111205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831812a4b0 con 0x7f83181e0e10 2024-09-16T03:49:57.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7efdf08ee640 1 --2- 172.21.15.70:0/2983094982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 0x7efdec1e83e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efde800c430 tx=0x7efde800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f831cb81640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83181e0e10 0x7f83181dcdf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83180aede0 tx=0x7f8310007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fe884045020 con 0x7fe8940aaf80 2024-09-16T03:49:57.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 client.?.objecter ms_handle_connect 0x7f83181e0e10 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fbb9f4f8640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fbb9f4f8640 10 monclient: _renew_subs 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fbb9f4f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb981d7590 con 0x7fbb84004800 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 10 client.4309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 15 client.4309.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 20 client.4309.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2f8007930 con 0x7fd2f8006e20 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fbb9f4f8640 1 librados: init done 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831000ac60 con 0x7f83181e0e10 2024-09-16T03:49:57.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: dump: 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.223 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8310018070 con 0x7f83181e0e10 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310013630 con 0x7f83181e0e10 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f8314ff9640 10 monclient: dump: 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.224 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f831d382640 5 monclient: authenticate success, global_id 4323 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f831d382640 10 monclient: _renew_subs 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f831d382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831812a720 con 0x7f83181e0e10 2024-09-16T03:49:57.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fe87bfff640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6804/198622594 conn(0x7fe88006dc00 0x7fe8800700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7efdddffb640 10 client.?.objecter ms_handle_connect 0x7efdec0af940 2024-09-16T03:49:57.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7efdddffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c36d9d640 1 --2- 172.21.15.70:0/3609293306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 0x7f3c30160420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c38074a00 tx=0x7f3c38069080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c26ffd640 10 client.?.objecter ms_handle_connect 0x7f3c300c6bb0 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.023+0000 7f3c26ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38095440 con 0x7f3c300c6bb0 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: dump: 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.227 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c380955e0 con 0x7f3c300c6bb0 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c380959b0 con 0x7f3c300c6bb0 2024-09-16T03:49:57.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c26ffd640 10 monclient: dump: 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7f3c27fff640 4 set_mon_vals no callback set 2024-09-16T03:49:57.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 5 monclient: authenticate success, global_id 4315 2024-09-16T03:49:57.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 10 monclient: _renew_subs 2024-09-16T03:49:57.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f83167fc640 4 set_mon_vals no callback set 2024-09-16T03:49:57.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 1 -- 172.21.15.70:0/3609293306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c300c8a70 con 0x7f3c300c6bb0 2024-09-16T03:49:57.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.024+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fd2fc01a070 con 0x7fd2f8006e20 2024-09-16T03:49:57.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 10 monclient: _renew_subs 2024-09-16T03:49:57.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 1 -- 172.21.15.70:0/3609293306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3002a4d0 con 0x7f3c300c6bb0 2024-09-16T03:49:57.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 15 client.4262.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f3c3759e640 1 librados: init done 2024-09-16T03:49:57.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7fe8937fe640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6804/198622594 conn(0x7fe88006dc00 0x7fe8800700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 20 client.4262.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7fe8937fe640 10 monclient: get_auth_request con 0x7fe88006dc00 auth_method 0 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f471359c640 1 -- 172.21.15.70:0/762772608 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f470c147440 con 0x7f470c14aa20 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f831d382640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f831d382640 10 monclient: _renew_subs 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f831d382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83181dd330 con 0x7f83181e0e10 2024-09-16T03:49:57.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7f831d382640 1 librados: init done 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8017070 con 0x7efdec0af940 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: dump: 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.235 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde80052f0 con 0x7efdec0af940 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 10 client.4311.objecter ms_handle_connect 0x7fe88006dc00 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe8937fe640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6804/198622594 conn(0x7fe88006dc00 0x7fe8800700c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0020e0 tx=0x7fe88c001460 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fbb9001a070 con 0x7fbb84004800 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 15 client.4318.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb981435f0 con 0x7fbb84004800 2024-09-16T03:49:57.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fbb8dffb640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6804/198622594 conn(0x7fbb7c06de00 0x7fbb7c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614016020 con 0x7f56180abbd0 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 10 monclient: dump: 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.238 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5614012870 con 0x7f56180abbd0 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f561f783640 5 monclient: authenticate success, global_id 4314 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f561f783640 10 monclient: _renew_subs 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f561f783640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f561f783640 1 -- 172.21.15.70:0/2770056566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56181d2c20 con 0x7f56180abbd0 2024-09-16T03:49:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8003830 con 0x7efdec0af940 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdddffb640 10 monclient: dump: 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efddf7fe640 4 set_mon_vals no callback set 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdf3353640 5 monclient: authenticate success, global_id 4321 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdf3353640 10 monclient: _renew_subs 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdf3353640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.025+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdec06fa10 con 0x7efdec0af940 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f4708045070 con 0x7f470c14aa20 2024-09-16T03:49:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f560dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.70:6804/198622594 conn(0x7fd2e406dcd0 0x7fd2e4070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fbb90056e40 con 0x7fbb84004800 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 10 client.4318.objecter ms_dispatch 0x7fbb98000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 3 client.4318.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 3 client.4318.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6804/198622594 conn(0x7fbb7c06de00 0x7fbb7c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb9ca93640 10 monclient: get_auth_request con 0x7fbb7c06de00 auth_method 0 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 10 client.4318.objecter checking command 1 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbb7c074810 0x7fbb7c076cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter _get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 15 client.4318.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter _get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 15 client.4318.objecter _session_command_op_assign 3 1 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 4 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 10 client.4318.objecter _send_command 1 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 --> v2:172.21.15.70:6805/3986044637 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb7c077340 con 0x7fbb7c074810 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb8dffb640 20 client.4318.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fd2fc08a2b0 con 0x7fd2f8006e20 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 10 client.4309.objecter ms_dispatch 0x7fd300000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 3 client.4309.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 3 client.4309.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd305b8a640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.70:6804/198622594 conn(0x7fd2e406dcd0 0x7fd2e4070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 15 client.4315.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 20 client.4315.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c300c2940 con 0x7f3c300c6bb0 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.70:6804/198622594 conn(0x7f46f006dc00 0x7f46f00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f47080ab790 con 0x7f470c14aa20 2024-09-16T03:49:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 10 client.4262.objecter ms_dispatch 0x7f470c000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 3 client.4262.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 3 client.4262.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 10 client.4262.objecter checking command 1 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.163:6804/1904387775 conn(0x7f46f0073be0 0x7f46f0076020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7efdf3353640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7efdf3353640 10 monclient: _renew_subs 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7efdf3353640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec12dcb0 con 0x7efdec0af940 2024-09-16T03:49:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7efdf3353640 1 librados: init done 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter _get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 15 client.4262.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter _get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 15 client.4262.objecter _session_command_op_assign 4 1 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 4 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 10 client.4262.objecter _send_command 1 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 --> v2:172.21.15.163:6804/1904387775 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f470802be30 con 0x7f46f0073be0 2024-09-16T03:49:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f4701ffb640 20 client.4262.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb9da95640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbb7c074810 0x7fbb7c076cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fbb9da95640 10 monclient: get_auth_request con 0x7fbb7c074810 auth_method 0 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fbb9da95640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbb7c074810 0x7fbb7c076cb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd305b8a640 10 monclient: get_auth_request con 0x7fd2e406dcd0 auth_method 0 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7fd2f6ffd640 10 client.4309.objecter checking command 1 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.163:6808/1625318059 conn(0x7fd2e4073cb0 0x7fd2e40760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter _get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 15 client.4309.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter _get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f560f7fe640 4 set_mon_vals no callback set 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 15 client.4309.objecter _session_command_op_assign 5 1 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 4 2024-09-16T03:49:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 10 client.4309.objecter _send_command 1 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 --> v2:172.21.15.163:6808/1625318059 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2e4076780 con 0x7fd2e4073cb0 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 20 client.4309.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc091050 con 0x7fd2f8006e20 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd2f6ffd640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fbb8dffb640 10 client.4318.objecter ms_handle_connect 0x7fbb7c074810 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbb900557d0 con 0x7fbb84004800 2024-09-16T03:49:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614012c60 con 0x7f56180abbd0 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f560dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fbb8dffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f560dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f560dffb640 10 monclient: dump: 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.251 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 15 client.4323.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f561f783640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83180ac3b0 con 0x7f83181e0e10 2024-09-16T03:49:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f83100137d0 con 0x7f83181e0e10 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f561f783640 10 monclient: _renew_subs 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fe884078990 con 0x7fe8940aaf80 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 10 client.4311.objecter ms_dispatch 0x7fe894000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 3 client.4311.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 3 client.4311.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f561f783640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f561f783640 1 -- 172.21.15.70:0/2770056566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5618071000 con 0x7f56180abbd0 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 10 client.4311.objecter checking command 1 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6809/2584301924 conn(0x7fe8800745c0 0x7fe880076a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter _get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 15 client.4311.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter _get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 15 client.4311.objecter _session_command_op_assign 2 1 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 4 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 10 client.4311.objecter _send_command 1 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 --> v2:172.21.15.70:6809/2584301924 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8800770f0 con 0x7fe8800745c0 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.026+0000 7fe87bfff640 20 client.4311.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fe893fff640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6809/2584301924 conn(0x7fe8800745c0 0x7fe880076a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe884077320 con 0x7fe8940aaf80 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fe87bfff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f4711338640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.70:6804/198622594 conn(0x7f46f006dc00 0x7f46f00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f4711338640 10 monclient: get_auth_request con 0x7f46f006dc00 auth_method 0 2024-09-16T03:49:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f8314ff9640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6804/198622594 conn(0x7f830406dc70 0x7f8304070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7fd306b8c640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.163:6808/1625318059 conn(0x7fd2e4073cb0 0x7fd2e40760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fbb9ca93640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6804/198622594 conn(0x7fbb7c06de00 0x7fbb7c0702c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb94035000 tx=0x7fbb9402d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47080b2050 con 0x7f470c14aa20 2024-09-16T03:49:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f4701ffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fbb8dffb640 10 client.4318.objecter ms_handle_connect 0x7fbb7c06de00 2024-09-16T03:49:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 15 client.4321.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efdec1e81a0 con 0x7efdec0af940 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.027+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f3c38095b50 con 0x7f3c300c6bb0 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f3c26ffd640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6804/198622594 conn(0x7f3c1806dcd0 0x7f3c18070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f3c3659c640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6804/198622594 conn(0x7f3c1806dcd0 0x7f3c18070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.028+0000 7f3c3659c640 10 monclient: get_auth_request con 0x7f3c1806dcd0 auth_method 0 2024-09-16T03:49:57.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f3c38106c40 con 0x7f3c300c6bb0 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 10 client.4315.objecter ms_dispatch 0x7f3c38000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 3 client.4315.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 3 client.4315.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 10 client.4315.objecter checking command 1 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6813/1499120614 conn(0x7f3c18073cb0 0x7f3c180760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter _get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 15 client.4315.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter _get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 15 client.4315.objecter _session_command_op_assign 1 1 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 4 2024-09-16T03:49:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 10 client.4315.objecter _send_command 1 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 --> v2:172.21.15.70:6813/1499120614 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c38092ec0 con 0x7f3c18073cb0 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 20 client.4315.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3c380d12e0 con 0x7f3c300c6bb0 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c26ffd640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fe893fff640 10 monclient: get_auth_request con 0x7fe8800745c0 auth_method 0 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8317fff640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6804/198622594 conn(0x7f830406dc70 0x7f8304070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8317fff640 10 monclient: get_auth_request con 0x7f830406dc70 auth_method 0 2024-09-16T03:49:57.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4710b37640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.163:6804/1904387775 conn(0x7f46f0073be0 0x7f46f0076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4710b37640 10 monclient: get_auth_request con 0x7f46f0073be0 auth_method 0 2024-09-16T03:49:57.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fd305b8a640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.70:6804/198622594 conn(0x7fd2e406dcd0 0x7fd2e4070190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0034000 tx=0x7fd2f002c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fd306b8c640 10 monclient: get_auth_request con 0x7fd2e4073cb0 auth_method 0 2024-09-16T03:49:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fd2f6ffd640 10 client.4309.objecter ms_handle_connect 0x7fd2e406dcd0 2024-09-16T03:49:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f831008b2a0 con 0x7f83181e0e10 2024-09-16T03:49:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4710b37640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.163:6804/1904387775 conn(0x7f46f0073be0 0x7f46f0076020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4701ffb640 10 client.4262.objecter ms_handle_connect 0x7f46f0073be0 2024-09-16T03:49:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 10 client.4323.objecter ms_dispatch 0x7f8318000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 3 client.4323.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c35d9b640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6813/1499120614 conn(0x7f3c18073cb0 0x7f3c180760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4711338640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.70:6804/198622594 conn(0x7f46f006dc00 0x7f46f00700c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009990 tx=0x7f46fc009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f5614019020 con 0x7f56180abbd0 2024-09-16T03:49:57.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 3 client.4323.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f3c35d9b640 10 monclient: get_auth_request con 0x7f3c18073cb0 auth_method 0 2024-09-16T03:49:57.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f4701ffb640 10 client.4262.objecter ms_handle_connect 0x7f46f006dc00 2024-09-16T03:49:57.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 10 client.4323.objecter checking command 1 2024-09-16T03:49:57.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f561f783640 1 librados: init done 2024-09-16T03:49:57.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7f8314ff9640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6800/1243681362 conn(0x7f8304073c50 0x7f8304076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8317fff640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6804/198622594 conn(0x7f830406dc70 0x7f8304070130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8308004650 tx=0x7f8308008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f3c3659c640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6804/198622594 conn(0x7f3c1806dcd0 0x7f3c18070190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c28034000 tx=0x7f3c2802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter _get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 15 client.4323.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter _get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 15 client.4323.objecter _session_command_op_assign 0 1 2024-09-16T03:49:57.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 4 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 10 client.4323.objecter _send_command 1 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 --> v2:172.21.15.70:6800/1243681362 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8310015e70 con 0x7f8304073c50 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 20 client.4323.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 10 client.4323.objecter ms_handle_connect 0x7f830406dc70 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8310092050 con 0x7f83181e0e10 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f8314ff9640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f3c26ffd640 10 client.4315.objecter ms_handle_connect 0x7f3c1806dcd0 2024-09-16T03:49:57.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fe893fff640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6809/2584301924 conn(0x7fe8800745c0 0x7fe880076a60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fe87bfff640 10 client.4311.objecter ms_handle_connect 0x7fe8800745c0 2024-09-16T03:49:57.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7efde801a070 con 0x7efdec0af940 2024-09-16T03:49:57.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f83177fe640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6800/1243681362 conn(0x7f8304073c50 0x7f8304076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f83177fe640 10 monclient: get_auth_request con 0x7f8304073c50 auth_method 0 2024-09-16T03:49:57.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== osd.3 v2:172.21.15.70:6805/3986044637 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb7c077340 con 0x7fbb7c074810 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fbb8dffb640 10 client.4318.objecter ms_dispatch 0x7fbb98000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fbb8dffb640 10 client.4318.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fbb8dffb640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 4 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fbb8dffb640 15 client.4318.objecter _session_command_op_remove 3 1 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.029+0000 7fd306b8c640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.163:6808/1625318059 conn(0x7fd2e4073cb0 0x7fd2e40760f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 10 client.4309.objecter ms_handle_connect 0x7fd2e4073cb0 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== osd.4 v2:172.21.15.163:6804/1904387775 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f470802be30 con 0x7f46f0073be0 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f4701ffb640 10 client.4262.objecter ms_dispatch 0x7f470c000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f4701ffb640 10 client.4262.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f4701ffb640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 4 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f4701ffb640 15 client.4262.objecter _session_command_op_remove 4 1 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f83177fe640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6800/1243681362 conn(0x7f8304073c50 0x7f8304076090 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f8314ff9640 10 client.4323.objecter ms_handle_connect 0x7f8304073c50 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f3c26ffd640 10 client.4315.objecter ms_handle_connect 0x7f3c18073cb0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7f3c35d9b640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6813/1499120614 conn(0x7f3c18073cb0 0x7f3c180760f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== osd.1 v2:172.21.15.70:6813/1499120614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c38092ec0 con 0x7f3c18073cb0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f3c26ffd640 10 client.4315.objecter ms_dispatch 0x7f3c38000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f3c26ffd640 10 client.4315.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== osd.5 v2:172.21.15.163:6808/1625318059 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2e4076780 con 0x7fd2e4073cb0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 10 client.4309.objecter ms_dispatch 0x7fd300000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 10 client.4309.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 4 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7fd2f6ffd640 15 client.4309.objecter _session_command_op_remove 5 1 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 10 client.4314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 15 client.4314.objecter _session_command_op_assign -1 1 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 20 client.4314.objecter _get_session osd=-1 returning homeless 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f561f783640 1 -- 172.21.15.70:0/2770056566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f56180ae6f0 con 0x7f56180abbd0 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.70:6804/198622594 conn(0x7f55fc06dcd0 0x7f55fc070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f561408a710 con 0x7f56180abbd0 2024-09-16T03:49:57.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 10 client.4314.objecter ms_dispatch 0x7f5618000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 3 client.4314.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 3 client.4314.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.70:6804/198622594 conn(0x7efdd006dcd0 0x7efdd0070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7efde808a0d0 con 0x7efdec0af940 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 10 client.4321.objecter ms_dispatch 0x7efdec000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 3 client.4321.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.030+0000 7efdddffb640 3 client.4321.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 10 client.4321.objecter checking command 1 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.163:6800/3710554613 conn(0x7efdd0073cb0 0x7efdd00760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter _get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 15 client.4321.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter _get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 15 client.4321.objecter _session_command_op_assign 7 1 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 4 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 10 client.4321.objecter _send_command 1 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 --> v2:172.21.15.163:6800/3710554613 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efde80054a0 con 0x7efdd0073cb0 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 20 client.4321.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efde8091050 con 0x7efdec0af940 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdddffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdf18f0640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.163:6800/3710554613 conn(0x7efdd0073cb0 0x7efdd00760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7efdf18f0640 10 monclient: get_auth_request con 0x7efdd0073cb0 auth_method 0 2024-09-16T03:49:57.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f3c26ffd640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 4 2024-09-16T03:49:57.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f3c26ffd640 15 client.4315.objecter _session_command_op_remove 1 1 2024-09-16T03:49:57.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 10 client.4314.objecter checking command 1 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.163:6812/2999981402 conn(0x7f55fc073cb0 0x7f55fc0760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter _get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 15 client.4314.objecter _session_command_op_remove -1 1 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter _get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 15 client.4314.objecter _session_command_op_assign 6 1 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 4 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 10 client.4314.objecter _send_command 1 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 --> v2:172.21.15.163:6812/2999981402 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5614012870 con 0x7f55fc073cb0 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 20 client.4314.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.031+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5614054ef0 con 0x7f56180abbd0 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f560dffb640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== osd.2 v2:172.21.15.70:6809/2584301924 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8800770f0 con 0x7fe8800745c0 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7fe87bfff640 10 client.4311.objecter ms_dispatch 0x7fe894000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7fe87bfff640 10 client.4311.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7fe87bfff640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 4 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7fe87bfff640 15 client.4311.objecter _session_command_op_remove 2 1 2024-09-16T03:49:57.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f561dd20640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.163:6812/2999981402 conn(0x7f55fc073cb0 0x7f55fc0760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f561dd20640 10 monclient: get_auth_request con 0x7f55fc073cb0 auth_method 0 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7efdf10ef640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.70:6804/198622594 conn(0x7efdd006dcd0 0x7efdd0070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7efdf10ef640 10 monclient: get_auth_request con 0x7efdd006dcd0 auth_method 0 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== osd.0 v2:172.21.15.70:6800/1243681362 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8310015e70 con 0x7f8304073c50 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f8314ff9640 10 client.4323.objecter ms_dispatch 0x7f8318000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f8314ff9640 10 client.4323.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f8314ff9640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 4 2024-09-16T03:49:57.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f8314ff9640 15 client.4323.objecter _session_command_op_remove 0 1 2024-09-16T03:49:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f561cd1e640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.70:6804/198622594 conn(0x7f55fc06dcd0 0x7f55fc070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f561cd1e640 10 monclient: get_auth_request con 0x7f55fc06dcd0 auth_method 0 2024-09-16T03:49:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f561dd20640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.163:6812/2999981402 conn(0x7f55fc073cb0 0x7f55fc0760f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7f560dffb640 10 client.4314.objecter ms_handle_connect 0x7f55fc073cb0 2024-09-16T03:49:57.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7efdf10ef640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.70:6804/198622594 conn(0x7efdd006dcd0 0x7efdd0070190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efde4009950 tx=0x7efde4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.032+0000 7efdddffb640 10 client.4321.objecter ms_handle_connect 0x7efdd006dcd0 2024-09-16T03:49:57.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdf18f0640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.163:6800/3710554613 conn(0x7efdd0073cb0 0x7efdd00760f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 10 client.4321.objecter ms_handle_connect 0x7efdd0073cb0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f561cd1e640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.70:6804/198622594 conn(0x7f55fc06dcd0 0x7f55fc070190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f561813c310 tx=0x7f5608008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 10 client.4314.objecter ms_handle_connect 0x7f55fc06dcd0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== osd.6 v2:172.21.15.163:6812/2999981402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5614012870 con 0x7f55fc073cb0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 10 client.4314.objecter ms_dispatch 0x7f5618000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 10 client.4314.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 4 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7f560dffb640 15 client.4314.objecter _session_command_op_remove 6 1 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== osd.7 v2:172.21.15.163:6800/3710554613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efde80054a0 con 0x7efdd0073cb0 2024-09-16T03:49:57.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 10 client.4321.objecter ms_dispatch 0x7efdec000b90 command_reply(tid 1: 0 ) 2024-09-16T03:49:57.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 10 client.4321.objecter _finish_command 1 = system:0 2024-09-16T03:49:57.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 4 2024-09-16T03:49:57.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.033+0000 7efdddffb640 15 client.4321.objecter _session_command_op_remove 7 1 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 15 client.4318.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 20 client.4318.objecter _get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 20 client.4318.objecter _get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 15 client.4318.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 20 client.4318.objecter get_session s=0x7fbb7c074160 osd=3 3 2024-09-16T03:49:57.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 15 client.4318.objecter _session_command_op_assign 3 2 2024-09-16T03:49:57.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 4 2024-09-16T03:49:57.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 10 client.4318.objecter _send_command 2 2024-09-16T03:49:57.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7fbb9f4f8640 1 -- 172.21.15.70:0/2595529247 --> v2:172.21.15.70:6805/3986044637 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb981430a0 con 0x7fbb7c074810 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 15 client.4262.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 20 client.4262.objecter _get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 20 client.4262.objecter _get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 15 client.4262.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 20 client.4262.objecter get_session s=0x7f46f0073530 osd=4 3 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 15 client.4262.objecter _session_command_op_assign 4 2 2024-09-16T03:49:57.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 4 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 10 client.4262.objecter _send_command 2 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.060+0000 7f471359c640 1 -- 172.21.15.70:0/762772608 --> v2:172.21.15.163:6804/1904387775 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f470c14b260 con 0x7f46f0073be0 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7fbb8dffb640 1 -- 172.21.15.70:0/2595529247 <== osd.3 v2:172.21.15.70:6805/3986044637 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbb981430a0 con 0x7fbb7c074810 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7fbb8dffb640 10 client.4318.objecter ms_dispatch 0x7fbb98000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7fbb8dffb640 10 client.4318.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7fbb8dffb640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 4 2024-09-16T03:49:57.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7fbb8dffb640 15 client.4318.objecter _session_command_op_remove 3 2 2024-09-16T03:49:57.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7f4701ffb640 1 -- 172.21.15.70:0/762772608 <== osd.4 v2:172.21.15.163:6804/1904387775 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f470c14b260 con 0x7f46f0073be0 2024-09-16T03:49:57.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7f4701ffb640 10 client.4262.objecter ms_dispatch 0x7f470c000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.061+0000 7f4701ffb640 10 client.4262.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 10 librados: watch_flush enter 2024-09-16T03:49:57.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 10 librados: watch_flush exit 2024-09-16T03:49:57.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 10 client.4318.objecter close_session for osd.3 2024-09-16T03:49:57.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbb7c074810 msgr2=0x7fbb7c076cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbb7c074810 0x7fbb7c076cb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 15 client.4311.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 20 client.4311.objecter _get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 20 client.4311.objecter _get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 15 client.4311.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 20 client.4311.objecter get_session s=0x7fe880073f10 osd=2 3 2024-09-16T03:49:57.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 15 client.4311.objecter _session_command_op_assign 2 2 2024-09-16T03:49:57.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f4701ffb640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 4 2024-09-16T03:49:57.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f4701ffb640 15 client.4262.objecter _session_command_op_remove 4 2 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 4 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 10 client.4311.objecter _send_command 2 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe898d2b640 1 -- 172.21.15.70:0/4219226857 --> v2:172.21.15.70:6809/2584301924 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8940af470 con 0x7fe8800745c0 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 10 librados: watch_flush enter 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 10 librados: watch_flush exit 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 10 client.4262.objecter close_session for osd.4 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 >> v2:172.21.15.163:6804/1904387775 conn(0x7f46f0073be0 msgr2=0x7f46f0076020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.163:6804/1904387775 conn(0x7f46f0073be0 0x7f46f0076020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 10 client.4309.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 15 client.4309.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 20 client.4309.objecter _get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 20 client.4309.objecter _get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 15 client.4309.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 20 client.4309.objecter get_session s=0x7fd2e4073600 osd=5 3 2024-09-16T03:49:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 15 client.4309.objecter _session_command_op_assign 5 2 2024-09-16T03:49:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 20 client.4262.objecter put_session s=0x7f46f0073530 osd=4 1 2024-09-16T03:49:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 4 2024-09-16T03:49:57.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 20 client.4262.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f46ef7fe640 10 client.4262.objecter successfully canceled tick 2024-09-16T03:49:57.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 10 client.4309.objecter _send_command 2 2024-09-16T03:49:57.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 --> v2:172.21.15.163:6808/1625318059 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2f80cb350 con 0x7fd2e4073cb0 2024-09-16T03:49:57.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe87bfff640 1 -- 172.21.15.70:0/4219226857 <== osd.2 v2:172.21.15.70:6809/2584301924 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe8940af470 con 0x7fe8800745c0 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe87bfff640 10 client.4311.objecter ms_dispatch 0x7fe894000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe87bfff640 10 client.4311.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe87bfff640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 4 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fe87bfff640 15 client.4311.objecter _session_command_op_remove 2 2 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 20 client.4318.objecter put_session s=0x7fbb7c074160 osd=3 1 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 20 client.4318.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 10 client.4318.objecter successfully canceled tick 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6804/198622594 conn(0x7fbb7c06de00 msgr2=0x7fbb7c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 --2- 172.21.15.70:0/2595529247 >> v2:172.21.15.70:6804/198622594 conn(0x7fbb7c06de00 0x7fbb7c0702c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb94035000 tx=0x7fbb9402d000 comp rx=0 tx=0).stop 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 10 monclient: shutdown 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 msgr2=0x7fbb9813daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 --2- 172.21.15.70:0/2595529247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb84004800 0x7fbb9813daa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000e5f0 tx=0x7fbb9000eab0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb9da95640 1 -- 172.21.15.70:0/2595529247 reap_dead start 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 shutdown_connections 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 >> 172.21.15.70:0/2595529247 conn(0x7fbb980b5300 msgr2=0x7fbb9813a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f831d382640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.062+0000 7f831d382640 15 client.4323.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 20 client.4323.objecter _get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 20 client.4323.objecter _get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 15 client.4323.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 20 client.4323.objecter get_session s=0x7f83040735a0 osd=0 3 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 15 client.4323.objecter _session_command_op_assign 0 2 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 4 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 10 client.4323.objecter _send_command 2 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 --> v2:172.21.15.70:6800/1243681362 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f83180af400 con 0x7f8304073c50 2024-09-16T03:49:57.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 15 client.4315.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 20 client.4315.objecter _get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 20 client.4315.objecter _get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 15 client.4315.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 20 client.4315.objecter get_session s=0x7f3c18073600 osd=1 3 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 15 client.4315.objecter _session_command_op_assign 1 2 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 4 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 10 client.4315.objecter _send_command 2 2024-09-16T03:49:57.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 --> v2:172.21.15.70:6813/1499120614 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c300b9720 con 0x7f3c18073cb0 2024-09-16T03:49:57.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd2f6ffd640 1 -- 172.21.15.70:0/3875884545 <== osd.5 v2:172.21.15.163:6808/1625318059 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd2f80cb350 con 0x7fd2e4073cb0 2024-09-16T03:49:57.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd2f6ffd640 10 client.4309.objecter ms_dispatch 0x7fd300000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd2f6ffd640 10 client.4309.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 10 librados: watch_flush enter 2024-09-16T03:49:57.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd2f6ffd640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 4 2024-09-16T03:49:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd2f6ffd640 15 client.4309.objecter _session_command_op_remove 5 2 2024-09-16T03:49:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 10 librados: watch_flush exit 2024-09-16T03:49:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 10 client.4311.objecter close_session for osd.2 2024-09-16T03:49:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6809/2584301924 conn(0x7fe8800745c0 msgr2=0x7fe880076a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6809/2584301924 conn(0x7fe8800745c0 0x7fe880076a60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 >> v2:172.21.15.70:6804/198622594 conn(0x7f46f006dc00 msgr2=0x7f46f00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 --2- 172.21.15.70:0/762772608 >> v2:172.21.15.70:6804/198622594 conn(0x7f46f006dc00 0x7f46f00700c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009990 tx=0x7f46fc009110 comp rx=0 tx=0).stop 2024-09-16T03:49:57.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f8314ff9640 1 -- 172.21.15.70:0/4122111205 <== osd.0 v2:172.21.15.70:6800/1243681362 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f83180af400 con 0x7f8304073c50 2024-09-16T03:49:57.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f8314ff9640 10 client.4323.objecter ms_dispatch 0x7f8318000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f8314ff9640 10 client.4323.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f8314ff9640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 4 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f8314ff9640 15 client.4323.objecter _session_command_op_remove 0 2 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 10 monclient: shutdown 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c14aa20 msgr2=0x7f470c1d8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 --2- 172.21.15.70:0/762772608 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f470c14aa20 0x7f470c1d8290 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4708031000 tx=0x7f47080046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c26ffd640 1 -- 172.21.15.70:0/3609293306 <== osd.1 v2:172.21.15.70:6813/1499120614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3c300b9720 con 0x7f3c18073cb0 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c26ffd640 10 client.4315.objecter ms_dispatch 0x7f3c38000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c26ffd640 10 client.4315.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c26ffd640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 4 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f3c26ffd640 15 client.4315.objecter _session_command_op_remove 1 2 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 20 client.4311.objecter put_session s=0x7fe880073f10 osd=2 1 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 20 client.4311.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 10 client.4311.objecter successfully canceled tick 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6804/198622594 conn(0x7fe88006dc00 msgr2=0x7fe8800700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 --2- 172.21.15.70:0/4219226857 >> v2:172.21.15.70:6804/198622594 conn(0x7fe88006dc00 0x7fe8800700c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0020e0 tx=0x7fe88c001460 comp rx=0 tx=0).stop 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 10 monclient: shutdown 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f4711b39640 1 -- 172.21.15.70:0/762772608 reap_dead start 2024-09-16T03:49:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fd30738d640 10 librados: watch_flush enter 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 msgr2=0x7fe8941e83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 --2- 172.21.15.70:0/4219226857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8940aaf80 0x7fe8941e83e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe88402a750 tx=0x7fe8840046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 shutdown_connections 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 shutdown_connections 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 >> 172.21.15.70:0/762772608 conn(0x7f470c0b1830 msgr2=0x7f470c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe893fff640 1 -- 172.21.15.70:0/4219226857 reap_dead start 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 10 librados: watch_flush enter 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 10 librados: watch_flush exit 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 10 client.4323.objecter close_session for osd.0 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6800/1243681362 conn(0x7f8304073c50 msgr2=0x7f8304076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6800/1243681362 conn(0x7f8304073c50 0x7f8304076090 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 20 client.4323.objecter put_session s=0x7f83040735a0 osd=0 1 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 20 client.4323.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 10 client.4323.objecter successfully canceled tick 2024-09-16T03:49:57.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 shutdown_connections 2024-09-16T03:49:57.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 >> 172.21.15.70:0/4219226857 conn(0x7fe894068050 msgr2=0x7fe8941414c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fbb7b7fe640 1 -- 172.21.15.70:0/2595529247 wait complete. 2024-09-16T03:49:57.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 shutdown_connections 2024-09-16T03:49:57.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.063+0000 7f46ef7fe640 1 -- 172.21.15.70:0/762772608 wait complete. 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 10 librados: watch_flush exit 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 10 client.4309.objecter close_session for osd.5 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 >> v2:172.21.15.163:6808/1625318059 conn(0x7fd2e4073cb0 msgr2=0x7fd2e40760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.163:6808/1625318059 conn(0x7fd2e4073cb0 0x7fd2e40760f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 20 client.4309.objecter put_session s=0x7fd2e4073600 osd=5 1 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 20 client.4309.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 10 client.4309.objecter successfully canceled tick 2024-09-16T03:49:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 10 librados: watch_flush enter 2024-09-16T03:49:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 10 librados: watch_flush exit 2024-09-16T03:49:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 shutdown_connections 2024-09-16T03:49:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 10 client.4315.objecter close_session for osd.1 2024-09-16T03:49:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6813/1499120614 conn(0x7f3c18073cb0 msgr2=0x7f3c180760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6813/1499120614 conn(0x7f3c18073cb0 0x7f3c180760f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fe879ffb640 1 -- 172.21.15.70:0/4219226857 wait complete. 2024-09-16T03:49:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 20 client.4315.objecter put_session s=0x7f3c18073600 osd=1 1 2024-09-16T03:49:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 >> v2:172.21.15.70:6804/198622594 conn(0x7fd2e406dcd0 msgr2=0x7fd2e4070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 --2- 172.21.15.70:0/3875884545 >> v2:172.21.15.70:6804/198622594 conn(0x7fd2e406dcd0 0x7fd2e4070190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0034000 tx=0x7fd2f002c000 comp rx=0 tx=0).stop 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6804/198622594 conn(0x7f830406dc70 msgr2=0x7f8304070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 --2- 172.21.15.70:0/4122111205 >> v2:172.21.15.70:6804/198622594 conn(0x7f830406dc70 0x7f8304070130 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8308004650 tx=0x7f8308008040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 10 monclient: shutdown 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83181e0e10 msgr2=0x7f83181dcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831d382640 1 --2- 172.21.15.70:0/4122111205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83181e0e10 0x7f83181dcdf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83180aede0 tx=0x7f8310007bd0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 20 client.4315.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 10 client.4315.objecter successfully canceled tick 2024-09-16T03:49:57.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f831cb81640 1 -- 172.21.15.70:0/4122111205 reap_dead start 2024-09-16T03:49:57.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 10 monclient: shutdown 2024-09-16T03:49:57.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fe879ffb640 1 librados: shutdown 2024-09-16T03:49:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 msgr2=0x7fd2f80c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 --2- 172.21.15.70:0/3875884545 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2f8006e20 0x7fd2f80c7b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc001380 tx=0x7fd2fc00cbe0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6804/198622594 conn(0x7f3c1806dcd0 msgr2=0x7f3c18070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 --2- 172.21.15.70:0/3609293306 >> v2:172.21.15.70:6804/198622594 conn(0x7f3c1806dcd0 0x7f3c18070190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c28034000 tx=0x7f3c2802e000 comp rx=0 tx=0).stop 2024-09-16T03:49:57.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 10 monclient: shutdown 2024-09-16T03:49:57.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 10 client.4314.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 msgr2=0x7f3c30160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 --2- 172.21.15.70:0/3609293306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c300c6bb0 0x7f3c30160420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3c38074a00 tx=0x7f3c38069080 comp rx=0 tx=0).stop 2024-09-16T03:49:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 15 client.4314.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 20 client.4314.objecter _get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f46ef7fe640 1 librados: shutdown 2024-09-16T03:49:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 20 client.4314.objecter _get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 15 client.4314.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c36d9d640 1 -- 172.21.15.70:0/3609293306 reap_dead start 2024-09-16T03:49:57.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 20 client.4314.objecter get_session s=0x7f55fc073600 osd=6 3 2024-09-16T03:49:57.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 15 client.4314.objecter _session_command_op_assign 6 2 2024-09-16T03:49:57.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 4 2024-09-16T03:49:57.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fbb7b7fe640 1 librados: shutdown 2024-09-16T03:49:57.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 10 client.4314.objecter _send_command 2 2024-09-16T03:49:57.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 shutdown_connections 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f561f783640 1 -- 172.21.15.70:0/2770056566 --> v2:172.21.15.163:6812/2999981402 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f56180ae6f0 con 0x7f55fc073cb0 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 >> 172.21.15.70:0/3609293306 conn(0x7f3c30005a40 msgr2=0x7f3c30037ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd306b8c640 1 -- 172.21.15.70:0/3875884545 reap_dead start 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 shutdown_connections 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.064+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 >> 172.21.15.70:0/3875884545 conn(0x7fd2f8005d50 msgr2=0x7fd2f8031170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 shutdown_connections 2024-09-16T03:49:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 >> 172.21.15.70:0/4122111205 conn(0x7f8318068050 msgr2=0x7f8318142f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 shutdown_connections 2024-09-16T03:49:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 shutdown_connections 2024-09-16T03:49:57.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 15 client.4321.objecter _session_command_op_assign -1 2 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 20 client.4321.objecter _get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7fd30738d640 1 -- 172.21.15.70:0/3875884545 wait complete. 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 20 client.4321.objecter _get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 15 client.4321.objecter _session_command_op_remove -1 2 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 20 client.4321.objecter get_session s=0x7efdd0073600 osd=7 3 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f3c24ff9640 1 -- 172.21.15.70:0/3609293306 wait complete. 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 15 client.4321.objecter _session_command_op_assign 7 2 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 4 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 10 client.4321.objecter _send_command 2 2024-09-16T03:49:57.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 --> v2:172.21.15.163:6800/3710554613 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efdec0684d0 con 0x7efdd0073cb0 2024-09-16T03:49:57.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 shutdown_connections 2024-09-16T03:49:57.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f831d382640 1 -- 172.21.15.70:0/4122111205 wait complete. 2024-09-16T03:49:57.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f3c24ff9640 1 librados: shutdown 2024-09-16T03:49:57.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7fd30738d640 1 librados: shutdown 2024-09-16T03:49:57.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f560dffb640 1 -- 172.21.15.70:0/2770056566 <== osd.6 v2:172.21.15.163:6812/2999981402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f56180ae6f0 con 0x7f55fc073cb0 2024-09-16T03:49:57.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f560dffb640 10 client.4314.objecter ms_dispatch 0x7f5618000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f560dffb640 10 client.4314.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f831d382640 1 librados: shutdown 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f560dffb640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 4 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f560dffb640 15 client.4314.objecter _session_command_op_remove 6 2 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdddffb640 1 -- 172.21.15.70:0/2983094982 <== osd.7 v2:172.21.15.163:6800/3710554613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efdec0684d0 con 0x7efdd0073cb0 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdddffb640 10 client.4321.objecter ms_dispatch 0x7efdec000b90 command_reply(tid 2: 0 ) 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdddffb640 10 client.4321.objecter _finish_command 2 = system:0 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdddffb640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 4 2024-09-16T03:49:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7efdddffb640 15 client.4321.objecter _session_command_op_remove 7 2 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.065+0000 7f56037fe640 10 librados: watch_flush enter 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 10 librados: watch_flush exit 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 10 client.4314.objecter close_session for osd.6 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 >> v2:172.21.15.163:6812/2999981402 conn(0x7f55fc073cb0 msgr2=0x7f55fc0760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.163:6812/2999981402 conn(0x7f55fc073cb0 0x7f55fc0760f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 20 client.4314.objecter put_session s=0x7f55fc073600 osd=6 1 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 20 client.4314.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 10 client.4314.objecter successfully canceled tick 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 10 librados: watch_flush enter 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 10 librados: watch_flush exit 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 10 client.4321.objecter close_session for osd.7 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 >> v2:172.21.15.163:6800/3710554613 conn(0x7efdd0073cb0 msgr2=0x7efdd00760f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.163:6800/3710554613 conn(0x7efdd0073cb0 0x7efdd00760f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 20 client.4321.objecter put_session s=0x7efdd0073600 osd=7 1 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 20 client.4321.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7efdf3353640 10 client.4321.objecter successfully canceled tick 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 >> v2:172.21.15.70:6804/198622594 conn(0x7f55fc06dcd0 msgr2=0x7f55fc070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 --2- 172.21.15.70:0/2770056566 >> v2:172.21.15.70:6804/198622594 conn(0x7f55fc06dcd0 0x7f55fc070190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f561813c310 tx=0x7f5608008040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 10 monclient: shutdown 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 msgr2=0x7f56180708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 --2- 172.21.15.70:0/2770056566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f56180abbd0 0x7f56180708e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f561400c460 tx=0x7f561400c920 comp rx=0 tx=0).stop 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f561dd20640 1 -- 172.21.15.70:0/2770056566 reap_dead start 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 shutdown_connections 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.066+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 >> 172.21.15.70:0/2770056566 conn(0x7f56180b1830 msgr2=0x7f561813e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 shutdown_connections 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7f56037fe640 1 -- 172.21.15.70:0/2770056566 wait complete. 2024-09-16T03:49:57.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7f56037fe640 1 librados: shutdown 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 >> v2:172.21.15.70:6804/198622594 conn(0x7efdd006dcd0 msgr2=0x7efdd0070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 1 --2- 172.21.15.70:0/2983094982 >> v2:172.21.15.70:6804/198622594 conn(0x7efdd006dcd0 0x7efdd0070190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efde4009950 tx=0x7efde4008040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 10 monclient: shutdown 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 msgr2=0x7efdec1e83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf3353640 1 --2- 172.21.15.70:0/2983094982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efdec0af940 0x7efdec1e83e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efde800c430 tx=0x7efde800c8f0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.067+0000 7efdf18f0640 1 -- 172.21.15.70:0/2983094982 reap_dead start 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.068+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 shutdown_connections 2024-09-16T03:49:57.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.068+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 >> 172.21.15.70:0/2983094982 conn(0x7efdec068020 msgr2=0x7efdec0b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.068+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 shutdown_connections 2024-09-16T03:49:57.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.068+0000 7efdf3353640 1 -- 172.21.15.70:0/2983094982 wait complete. 2024-09-16T03:49:57.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.068+0000 7efdf3353640 1 librados: shutdown 2024-09-16T03:49:57.332 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:57.333 INFO:teuthology.orchestra.run.smithi070.stdout:64424509444 2024-09-16T03:49:57.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:49:57.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:49:57.335 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:57.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:49:57.336 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:57.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:49:57.337 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:57.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:49:57.338 INFO:teuthology.orchestra.run.smithi070.stdout:55834574853 2024-09-16T03:49:57.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:49:57.339 INFO:teuthology.orchestra.run.smithi070.stdout:64424509445 2024-09-16T03:49:57.339 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:49:57.340 INFO:teuthology.orchestra.run.smithi070.stdout:60129542148 2024-09-16T03:49:57.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7f8d99e8f640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7f8d99e8f640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7f8d99e8f640 10 monclient: monmap: 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.539667+0000 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.539667+0000 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.541 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7fd48759e640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7fd48759e640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.538+0000 7fd48759e640 10 monclient: monmap: 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.540088+0000 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.540088+0000 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.543 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7fd48759e640 1 Processor -- start 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7fd48759e640 1 -- start start 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7fd48759e640 10 monclient: init 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7f8d99e8f640 1 Processor -- start 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7f8d99e8f640 1 -- start start 2024-09-16T03:49:57.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.539+0000 7f8d99e8f640 10 monclient: init 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd480006e30 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd480034b90 0x7fd480030480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-c con 0x7fd480034b90 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4800caa20 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd480035740 con 0x7fd480006e30 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4800cae30 con 0x7fd4800caa20 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fd480007210 con 0x7fd480034b90 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fd48759e640 10 monclient(hunting): authenticate will time out at 756.202952s 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d8c006de0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d8c0302f0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d8c0332f0 0x7f8d8c0337a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8d8c0332f0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8c035f40 con 0x7f8d8c0302f0 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8d8c0071c0 con 0x7f8d8c0332f0 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8c0353a0 con 0x7f8d8c006de0 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d99e8f640 10 monclient(hunting): authenticate will time out at 756.203121s 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fb69b0af640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fb69b0af640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.540+0000 7fb69b0af640 10 monclient: monmap: 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.547 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.542210+0000 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.542210+0000 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.548 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd480034b90 0x7fd480030480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd480034b90 0x7fd480030480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34796/0 (socket says 172.21.15.70:34796) 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 1 -- 172.21.15.70:0/1984141785 learned_addr learned my addr 172.21.15.70:0/1984141785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 10 monclient(hunting): get_auth_request con 0x7fd480034b90 auth_method 0 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fd485d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45762/0 (socket says 172.21.15.70:45762) 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 1 -- 172.21.15.70:0/3468186838 learned_addr learned my addr 172.21.15.70:0/3468186838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): get_auth_request con 0x7f8d8c006de0 auth_method 0 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fb69b0af640 1 Processor -- start 2024-09-16T03:49:57.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fb69b0af640 1 -- start start 2024-09-16T03:49:57.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fb69b0af640 10 monclient: init 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d937fe640 1 -- 172.21.15.70:0/3468186838 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609869139 0 0) 0x7f8d8c035f40 con 0x7f8d8c0302f0 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 1 -- 172.21.15.70:0/3468186838 >> v2:172.21.15.70:3300/0 conn(0x7f8d8c0332f0 msgr2=0x7f8d8c0337a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 1 --2- 172.21.15.70:0/3468186838 >> v2:172.21.15.70:3300/0 conn(0x7f8d8c0332f0 0x7f8d8c0337a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 1 -- 172.21.15.70:0/3468186838 >> v1:172.21.15.163:6789/0 conn(0x7f8d8c0302f0 legacy=0x7f8d8c035830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fe80c023640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fe80c023640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.541+0000 7fe80c023640 10 monclient: monmap: 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.543237+0000 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.543237+0000 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.553 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.554 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.554 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d98e8d640 1 -- 172.21.15.70:0/3468186838 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c036d70 con 0x7f8d8c006de0 2024-09-16T03:49:57.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7f8d937fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:57.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fe65932a640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fe65932a640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fe65932a640 10 monclient: monmap: 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.544157+0000 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.544157+0000 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.556 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2024-09-16T03:49:57.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 1 -- 172.21.15.70:0/1984141785 >> v1:172.21.15.163:6789/0 conn(0x7fd4800caa20 legacy=0x7fd4800332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 1 -- 172.21.15.70:0/1984141785 >> v1:172.21.15.70:6789/0 conn(0x7fd480006e30 legacy=0x7fd480035c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.542+0000 7fd485d9b640 1 -- 172.21.15.70:0/1984141785 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4800339f0 con 0x7fd480034b90 2024-09-16T03:49:57.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe80c023640 1 Processor -- start 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6940aaf50 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-b con 0x7fb6940aa780 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb694072e80 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7feef2e12640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7feef2e12640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7feef2e12640 10 monclient: monmap: 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.545328+0000 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.545328+0000 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.560 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6940aebe0 con 0x7fb694072e80 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6940ab7e0 con 0x7fb6940aaf50 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fb69414b580 con 0x7fb6940aa780 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb69b0af640 10 monclient(hunting): authenticate will time out at 756.205239s 2024-09-16T03:49:57.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45774/0 (socket says 172.21.15.70:45774) 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 1 -- 172.21.15.70:0/245731737 learned_addr learned my addr 172.21.15.70:0/245731737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): get_auth_request con 0x7fb6940aa780 auth_method 0 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fb6998ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe65932a640 1 Processor -- start 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe65932a640 1 -- start start 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe65932a640 10 monclient: init 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7f7c2e586640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7f7c2e586640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7f7c2e586640 10 monclient: monmap: 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.563 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.545064+0000 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.545064+0000 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.564 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.565 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 1 -- 172.21.15.70:0/245731737 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248316080 0 0) 0x7fb6940ab7e0 con 0x7fb6940aaf50 2024-09-16T03:49:57.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 10 monclient(hunting): _negotiate 2024-09-16T03:49:57.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 10 monclient(hunting): my global_id is 4336 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6990ab640 1 -- 172.21.15.70:0/245731737 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb680003520 con 0x7fb6940aaf50 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 1 -- 172.21.15.70:0/245731737 >> v1:172.21.15.163:6789/0 conn(0x7fb6940aaf50 legacy=0x7fb6940ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 1 -- 172.21.15.70:0/245731737 >> v1:172.21.15.70:6789/0 conn(0x7fb694072e80 legacy=0x7fb69414ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7f8d98e8d640 1 --2- 172.21.15.70:0/3468186838 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009870 tx=0x7f8d8002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e96c1610c88c93d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fb6998ac640 1 -- 172.21.15.70:0/245731737 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6940ab330 con 0x7fb6940aa780 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe80c023640 1 -- start start 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.543+0000 7fe80c023640 10 monclient: init 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fe804142700 0x7fe804142f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-c con 0x7fe804142700 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8040ac0f0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 0x7fe80413c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-b con 0x7fe80413c1c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8040ab600 con 0x7fe8040ac0f0 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fe804142ae0 con 0x7fe804142700 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fe8040affa0 con 0x7fe80413c1c0 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80c023640 10 monclient(hunting): authenticate will time out at 756.206494s 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7f416a3d4640 10 monclient: get_monmap_and_config 2024-09-16T03:49:57.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7f416a3d4640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7f416a3d4640 10 monclient: monmap: 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:57.545731+0000 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:57.545731+0000 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:57.569 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 0x7fe64c0c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-c con 0x7fe64c006b40 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe8095be640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe8040ac0f0 0x7fe8040af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33710/0 (socket says 172.21.15.70:33710) 2024-09-16T03:49:57.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe8095be640 1 -- 172.21.15.70:0/341080098 learned_addr learned my addr 172.21.15.70:0/341080098 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fd485d9b640 1 --2- 172.21.15.70:0/1984141785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480034b90 0x7fd480030480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd480033c80 tx=0x7fd48806aac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3580798a3555bbb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe64c006f20 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe64c031980 0x7fe64c02f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-a con 0x7fe64c031980 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0c7d40 con 0x7fe64c006f20 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe64c0324c0 con 0x7fe64c031980 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fe64c02f6d0 con 0x7fe64c006b40 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe65932a640 10 monclient(hunting): authenticate will time out at 756.206862s 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80a5c0640 1 --2- 172.21.15.70:0/341080098 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 0x7fe80413c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80a5c0640 10 monclient(hunting): get_auth_request con 0x7fe80413c1c0 auth_method 0 2024-09-16T03:49:57.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80a5c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80a5c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7fe80a5c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe64c031980 0x7fe64c02f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe64c031980 0x7fe64c02f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49128/0 (socket says 172.21.15.70:49128) 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 -- 172.21.15.70:0/478702335 learned_addr learned my addr 172.21.15.70:0/478702335 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe64c031980 auth_method 0 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 1 --2- 172.21.15.70:0/341080098 >> v2:172.21.15.163:3300/0 conn(0x7fe804142700 0x7fe804142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.544+0000 7f7c2e586640 1 Processor -- start 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7f7c2e586640 1 -- start start 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7f7c2e586640 10 monclient: init 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 1 --2- 172.21.15.70:0/478702335 >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 0x7fe64c0c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): get_auth_request con 0x7fe64c006b40 auth_method 0 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7f416a3d4640 1 Processor -- start 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7f416a3d4640 1 -- start start 2024-09-16T03:49:57.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7f416a3d4640 10 monclient: init 2024-09-16T03:49:57.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): get_auth_request con 0x7fe804142700 auth_method 0 2024-09-16T03:49:57.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe809dbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 1 -- 172.21.15.70:0/341080098 >> v2:172.21.15.163:3300/0 conn(0x7fe804142700 msgr2=0x7fe804142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 1 --2- 172.21.15.70:0/341080098 >> v2:172.21.15.163:3300/0 conn(0x7fe804142700 0x7fe804142f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 1 -- 172.21.15.70:0/341080098 >> v1:172.21.15.70:6789/0 conn(0x7fe8040ac0f0 legacy=0x7fe8040af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe80a5c0640 1 -- 172.21.15.70:0/341080098 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8040ad050 con 0x7fe80413c1c0 2024-09-16T03:49:57.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-09-16T03:49:57.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 1 -- 172.21.15.70:0/1984141785 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4880b3070 con 0x7fd480034b90 2024-09-16T03:49:57.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 -- 172.21.15.70:0/478702335 >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 msgr2=0x7fe64c0c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: dump: 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 1 -- 172.21.15.70:0/1984141785 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4880ae440 con 0x7fd480034b90 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 1 -- 172.21.15.70:0/1984141785 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4880ae830 con 0x7fd480034b90 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fd48559a640 10 monclient: dump: 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.584 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.585 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.585 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.585 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.585 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.585 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 --2- 172.21.15.70:0/478702335 >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 0x7fe64c0c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 -- 172.21.15.70:0/478702335 >> v1:172.21.15.163:6789/0 conn(0x7fe64c006f20 legacy=0x7fe64c0329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe653fff640 1 -- 172.21.15.70:0/478702335 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c02f880 con 0x7fe64c031980 2024-09-16T03:49:57.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6998ac640 1 --2- 172.21.15.70:0/245731737 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098a0 tx=0x7fb68402a1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba6882b35d48da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b28f0 0x7f7c28070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-b con 0x7f7c280b28f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7c28141940 0x7f7c28071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28141940 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c280b30f0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c281411a0 con 0x7f7c280b30f0 2024-09-16T03:49:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7c280b2cd0 con 0x7f7c28141940 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2806dc20 con 0x7f7c280b28f0 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c2e586640 10 monclient(hunting): authenticate will time out at 756.208175s 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7fe652ffd640 1 --2- 172.21.15.70:0/478702335 >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 0x7fe64c0c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 1 -- 172.21.15.70:0/3468186838 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80037070 con 0x7f8d8c006de0 2024-09-16T03:49:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: dump: 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d99e8f640 5 monclient: authenticate success, global_id 4274 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d99e8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 1 -- 172.21.15.70:0/3468186838 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d80032440 con 0x7f8d8c006de0 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d99e8f640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f8d99e8f640 4 set_mon_vals no callback set 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:49:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f41640aaa40 0x7f41640af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-c con 0x7f41640aaa40 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41640af940 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f4164149a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4164142f50 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41640aba90 con 0x7f41640af940 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f41640aae20 con 0x7f41640aaa40 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f41640736d0 con 0x7f4164142f50 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f416a3d4640 10 monclient(hunting): authenticate will time out at 756.208427s 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7c28141940 0x7f7c28071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7c28141940 0x7f7c28071280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49138/0 (socket says 172.21.15.70:49138) 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2650237154 learned_addr learned my addr 172.21.15.70:0/2650237154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request con 0x7f7c28141940 auth_method 0 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 1 -- 172.21.15.70:0/245731737 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684037070 con 0x7fb6940aa780 2024-09-16T03:49:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f41637fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f41640af940 0x7f4164142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:36048/0 (socket says 172.21.15.70:36048) 2024-09-16T03:49:57.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f41637fe640 1 -- 172.21.15.70:0/806119802 learned_addr learned my addr 172.21.15.70:0/806119802 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:57.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f4163fff640 1 --2- 172.21.15.70:0/806119802 >> v2:172.21.15.163:3300/0 conn(0x7f41640aaa40 0x7f41640af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 10 monclient: dump: 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.595 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7f4168971640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f4164149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fb6990ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb69b0af640 5 monclient: authenticate success, global_id 4277 2024-09-16T03:49:57.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb69b0af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): get_auth_request con 0x7f41640aaa40 auth_method 0 2024-09-16T03:49:57.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 1 -- 172.21.15.70:0/245731737 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb684032440 con 0x7fb6940aa780 2024-09-16T03:49:57.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f8d99e8f640 10 monclient: shutdown 2024-09-16T03:49:57.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f8d99e8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f8d99e8f640 1 -- 172.21.15.70:0/3468186838 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 msgr2=0x7f8d8c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f8d99e8f640 1 --2- 172.21.15.70:0/3468186838 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009870 tx=0x7f8d8002a270 comp rx=0 tx=0).stop 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): get_auth_request con 0x7f4164142f50 auth_method 0 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7feef2e12640 1 Processor -- start 2024-09-16T03:49:57.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7feef2e12640 1 -- start start 2024-09-16T03:49:57.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.545+0000 7feef2e12640 10 monclient: init 2024-09-16T03:49:57.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b28f0 msgr2=0x7f7c28070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 1 --2- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b28f0 0x7f7c28070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7feeec0aaa40 0x7feeec0af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-c con 0x7feeec0aaa40 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feeec073680 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feeec0af940 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2650237154 >> v1:172.21.15.70:6789/0 conn(0x7f7c280b30f0 legacy=0x7f7c2813df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeec143740 con 0x7feeec073680 2024-09-16T03:49:57.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeec0aae20 con 0x7feeec0af940 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7feeec0aaa40 0x7feeec0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7feeec0aaa40 0x7feeec0af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34814/0 (socket says 172.21.15.70:34814) 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 1 -- 172.21.15.70:0/171762738 learned_addr learned my addr 172.21.15.70:0/171762738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7feeec0ab160 con 0x7feeec0aaa40 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef2e12640 10 monclient(hunting): authenticate will time out at 756.208815s 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 10 monclient(hunting): get_auth_request con 0x7feeec0aaa40 auth_method 0 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7feef1e10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2650237154 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c280735a0 con 0x7f7c28141940 2024-09-16T03:49:57.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 1 -- 172.21.15.70:0/806119802 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 msgr2=0x7f4164149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f4164149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 1 -- 172.21.15.70:0/806119802 >> v1:172.21.15.163:6789/0 conn(0x7f41640af940 legacy=0x7f4164142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4162ffd640 1 -- 172.21.15.70:0/806119802 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3209738782 0 0) 0x7f41640aba90 con 0x7f41640af940 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4163fff640 1 -- 172.21.15.70:0/806119802 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41640ac9b0 con 0x7f41640aaa40 2024-09-16T03:49:57.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7f4168971640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f4164149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb69b0af640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb69b0af640 4 set_mon_vals no callback set 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 1 -- 172.21.15.70:0/245731737 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684037070 con 0x7fb6940aa780 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fb6990ab640 10 monclient: dump: 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.610 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.611 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fd48759e640 5 monclient: authenticate success, global_id 4327 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fd48759e640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fd48759e640 4 set_mon_vals no callback set 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fd48759e640 10 monclient: shutdown 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fd48759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fd48759e640 1 -- 172.21.15.70:0/1984141785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480034b90 msgr2=0x7fd480030480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fd48759e640 1 --2- 172.21.15.70:0/1984141785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480034b90 0x7fd480030480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd480033c80 tx=0x7fd48806aac0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient(hunting): handle_auth_done global_id 4351 payload 274 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 1 -- 172.21.15.70:0/171762738 >> v1:172.21.15.163:6789/0 conn(0x7feeec0af940 legacy=0x7feeec143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef0e0e640 1 -- 172.21.15.70:0/171762738 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295307238 0 0) 0x7feeec0aae20 con 0x7feeec0af940 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 1 -- 172.21.15.70:0/171762738 >> v1:172.21.15.70:6789/0 conn(0x7feeec073680 legacy=0x7feeec142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7feef1e10640 1 -- 172.21.15.70:0/171762738 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec0acc90 con 0x7feeec0aaa40 2024-09-16T03:49:57.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7feef0e0e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 10 monclient: shutdown 2024-09-16T03:49:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 -- 172.21.15.70:0/245731737 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 msgr2=0x7fb69414a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 --2- 172.21.15.70:0/245731737 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098a0 tx=0x7fb68402a1f0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fd48759e640 1 -- 172.21.15.70:0/1984141785 shutdown_connections 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fd48759e640 1 --2- 172.21.15.70:0/1984141785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480034b90 0x7fd480030480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fd48759e640 1 -- 172.21.15.70:0/1984141785 >> 172.21.15.70:0/1984141785 conn(0x7fd480005d60 msgr2=0x7fd480006170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fd48759e640 1 -- 172.21.15.70:0/1984141785 shutdown_connections 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fd48759e640 1 -- 172.21.15.70:0/1984141785 wait complete. 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 -- 172.21.15.70:0/3468186838 shutdown_connections 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 --2- 172.21.15.70:0/3468186838 >> v2:172.21.15.70:3300/0 conn(0x7f8d8c0332f0 0x7f8d8c0337a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 --2- 172.21.15.70:0/3468186838 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c006de0 0x7f8d8c0ceef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 -- 172.21.15.70:0/3468186838 >> 172.21.15.70:0/3468186838 conn(0x7f8d8c005d60 msgr2=0x7f8d8c006170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 -- 172.21.15.70:0/245731737 shutdown_connections 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 --2- 172.21.15.70:0/245731737 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb6940aa780 0x7fb69414a930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 -- 172.21.15.70:0/245731737 >> 172.21.15.70:0/245731737 conn(0x7fb6940b46e0 msgr2=0x7fb6940b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 -- 172.21.15.70:0/245731737 shutdown_connections 2024-09-16T03:49:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe80a5c0640 1 --2- 172.21.15.70:0/341080098 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 0x7fe80413c5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00cc60 tx=0x7fe7fc02b380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65f32c2b67e6763f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c1ffff640 1 --2- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c28071280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94361fbc8a1b631 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.546+0000 7fe653fff640 1 --2- 172.21.15.70:0/478702335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c02f190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe654074660 tx=0x7fe65406aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0770bee6d88473c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f4162ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:57.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 -- 172.21.15.70:0/3468186838 shutdown_connections 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 1 -- 172.21.15.70:0/341080098 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc007bc0 con 0x7fe80413c1c0 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: dump: 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 1 -- 172.21.15.70:0/341080098 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7fc007d60 con 0x7fe80413c1c0 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 1 -- 172.21.15.70:0/2650237154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18037070 con 0x7f7c28141940 2024-09-16T03:49:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 1 -- 172.21.15.70:0/341080098 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc0092f0 con 0x7fe80413c1c0 2024-09-16T03:49:57.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7feef1e10640 1 --2- 172.21.15.70:0/171762738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0aaa40 0x7feeec0af400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feee8009870 tx=0x7feee802e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=286e50b623312361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe808dbd640 10 monclient: dump: 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.622 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 10 monclient: dump: 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe80c023640 5 monclient: authenticate success, global_id 4280 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe80c023640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe80c023640 4 set_mon_vals no callback set 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f7c277fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 1 -- 172.21.15.70:0/2650237154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18032440 con 0x7f7c28141940 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 1 -- 172.21.15.70:0/2650237154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18032810 con 0x7f7c28141940 2024-09-16T03:49:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4163fff640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41640af400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a8c0 tx=0x7f415c02a190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b699bcd574285db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c277fe640 10 monclient: dump: 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7f8d99e8f640 1 -- 172.21.15.70:0/3468186838 wait complete. 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 1 -- 172.21.15.70:0/478702335 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6540af7d0 con 0x7fe64c031980 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: dump: 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.628 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 1 -- 172.21.15.70:0/171762738 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee803a070 con 0x7feeec0aaa40 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe65932a640 5 monclient: authenticate success, global_id 4326 2024-09-16T03:49:57.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe65932a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 1 -- 172.21.15.70:0/478702335 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6540aae50 con 0x7fe64c031980 2024-09-16T03:49:57.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe651ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe65932a640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.547+0000 7fe65932a640 4 set_mon_vals no callback set 2024-09-16T03:49:57.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 10 monclient: shutdown 2024-09-16T03:49:57.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 -- 172.21.15.70:0/478702335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 msgr2=0x7fe64c02f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe651ffb640 1 -- 172.21.15.70:0/478702335 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6540afc30 con 0x7fe64c031980 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: dump: 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.632 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 1 -- 172.21.15.70:0/171762738 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8033710 con 0x7feeec0aaa40 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef2e12640 5 monclient: authenticate success, global_id 4351 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef2e12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 1 -- 172.21.15.70:0/171762738 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8033ae0 con 0x7feeec0aaa40 2024-09-16T03:49:57.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 --2- 172.21.15.70:0/478702335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c02f190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe654074660 tx=0x7fe65406aac0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe651ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef0e0e640 10 monclient: dump: 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.634 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef2e12640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7feef2e12640 4 set_mon_vals no callback set 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fb69b0af640 1 -- 172.21.15.70:0/245731737 wait complete. 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 10 monclient: monmap: 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 10 monclient: monmap: 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.637 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 -- 172.21.15.70:0/478702335 shutdown_connections 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 --2- 172.21.15.70:0/478702335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c02f190 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 --2- 172.21.15.70:0/478702335 >> v2:172.21.15.163:3300/0 conn(0x7fe64c006b40 0x7fe64c0c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 -- 172.21.15.70:0/478702335 >> 172.21.15.70:0/478702335 conn(0x7fe64c005c30 msgr2=0x7fe64c006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 -- 172.21.15.70:0/478702335 shutdown_connections 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.548+0000 7fe65932a640 1 -- 172.21.15.70:0/478702335 wait complete. 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 librados: starting msgr at 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 librados: starting objecter 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 Processor -- start 2024-09-16T03:49:57.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 -- start start 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 librados: setting wanted keys 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 1 librados: calling monclient init 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fd48759e640 10 monclient: init 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f8d99e8f640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f8d99e8f640 10 monclient: monmap: 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.639 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 librados: starting msgr at 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 librados: starting objecter 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 Processor -- start 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 10 monclient: shutdown 2024-09-16T03:49:57.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0aaa40 msgr2=0x7feeec0af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 1 --2- 172.21.15.70:0/171762738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0aaa40 0x7feeec0af400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feee8009870 tx=0x7feee802e6c0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 shutdown_connections 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 1 --2- 172.21.15.70:0/171762738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0aaa40 0x7feeec0af400 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 >> 172.21.15.70:0/171762738 conn(0x7feeec068470 msgr2=0x7feeec0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 librados: starting msgr at 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 librados: starting objecter 2024-09-16T03:49:57.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 Processor -- start 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe80c023640 10 monclient: shutdown 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe80c023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe80c023640 1 -- 172.21.15.70:0/341080098 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 msgr2=0x7fe80413c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe80c023640 1 --2- 172.21.15.70:0/341080098 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 0x7fe80413c5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00cc60 tx=0x7fe7fc02b380 comp rx=0 tx=0).stop 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 -- 172.21.15.70:0/341080098 shutdown_connections 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 --2- 172.21.15.70:0/341080098 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe80413c1c0 0x7fe80413c5d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 --2- 172.21.15.70:0/341080098 >> v2:172.21.15.163:3300/0 conn(0x7fe804142700 0x7fe804142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 -- 172.21.15.70:0/341080098 >> 172.21.15.70:0/341080098 conn(0x7fe8040b1650 msgr2=0x7fe8040b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe65932a640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fe65932a640 10 monclient: monmap: 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.643 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c2e586640 5 monclient: authenticate success, global_id 4332 2024-09-16T03:49:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c2e586640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f7c2e586640 4 set_mon_vals no callback set 2024-09-16T03:49:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 librados: starting msgr at 2024-09-16T03:49:57.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 librados: starting objecter 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f7c2e586640 10 monclient: shutdown 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f7c2e586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f7c2e586640 1 -- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 msgr2=0x7f7c28071280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c28071280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009870 tx=0x7f7c1802a130 comp rx=0 tx=0).stop 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 -- start start 2024-09-16T03:49:57.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 Processor -- start 2024-09-16T03:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 -- 172.21.15.70:0/341080098 shutdown_connections 2024-09-16T03:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 -- start start 2024-09-16T03:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 librados: setting wanted keys 2024-09-16T03:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 1 librados: calling monclient init 2024-09-16T03:49:57.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe65932a640 10 monclient: init 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 librados: setting wanted keys 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 1 librados: calling monclient init 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7fb69b0af640 10 monclient: init 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-c con 0x7fb694072e80 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 0x7fb69413f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-a con 0x7fb6940aa780 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 0x7fb694144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): picked mon.noname-b con 0x7fb6940aaf50 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb694152d60 con 0x7fb6940aa780 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb694152be0 con 0x7fb6940aaf50 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fb694152ee0 con 0x7fb694072e80 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fb69b0af640 10 monclient(hunting): authenticate will time out at 756.212771s 2024-09-16T03:49:57.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 1 -- 172.21.15.70:0/806119802 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c037070 con 0x7f41640aaa40 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: dump: 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.649 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f416a3d4640 5 monclient: authenticate success, global_id 4354 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f416a3d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 1 -- 172.21.15.70:0/806119802 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f415c032440 con 0x7f41640aaa40 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f4162ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f416a3d4640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.549+0000 7f416a3d4640 4 set_mon_vals no callback set 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 10 monclient: shutdown 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 -- 172.21.15.70:0/806119802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 msgr2=0x7f41640af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f4162ffd640 1 -- 172.21.15.70:0/806119802 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c037070 con 0x7f41640aaa40 2024-09-16T03:49:57.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41640af400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f415c00a8c0 tx=0x7f415c02a190 comp rx=0 tx=0).stop 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f4162ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 -- 172.21.15.70:0/806119802 shutdown_connections 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f4164149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 --2- 172.21.15.70:0/806119802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41640af400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 -- 172.21.15.70:0/806119802 >> 172.21.15.70:0/806119802 conn(0x7f4164068020 msgr2=0x7f41640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 -- 172.21.15.70:0/806119802 shutdown_connections 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f416a3d4640 1 -- 172.21.15.70:0/806119802 wait complete. 2024-09-16T03:49:57.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7fe80c023640 1 -- 172.21.15.70:0/341080098 wait complete. 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 0x7fd48002fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-b con 0x7fd480006e30 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 shutdown_connections 2024-09-16T03:49:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 -- start start 2024-09-16T03:49:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 librados: setting wanted keys 2024-09-16T03:49:57.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 1 librados: calling monclient init 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.550+0000 7f8d99e8f640 10 monclient: init 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45810/0 (socket says 172.21.15.70:45810) 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 1 -- 172.21.15.70:0/1666014029 learned_addr learned my addr 172.21.15.70:0/1666014029 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): get_auth_request con 0x7fb694072e80 auth_method 0 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 0x7fb694144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f8d99e8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f8d99e8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f8d99e8f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 0x7f8d8c0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8d8c006de0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f8d99e8f640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c0302f0 0x7f8d8c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): get_auth_request con 0x7fb6940aaf50 auth_method 0 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd485d9b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 0x7fd48002fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 0x7fd48002fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34834/0 (socket says 172.21.15.70:34834) 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd485d9b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45816/0 (socket says 172.21.15.70:45816) 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48659c640 1 -- 172.21.15.70:0/385473594 learned_addr learned my addr 172.21.15.70:0/385473594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-c con 0x7fd480034b90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4800caa20 0x7fd48002dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe65932a640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe65932a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe65932a640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a0ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f7c2e586640 1 -- 172.21.15.70:0/2650237154 shutdown_connections 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c28071280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2650237154 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b28f0 0x7f7c28070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f7c2e586640 1 -- 172.21.15.70:0/2650237154 >> 172.21.15.70:0/2650237154 conn(0x7f7c280b3770 msgr2=0x7f7c280b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb69a8ae640 1 -- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 msgr2=0x7fb694147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fb6998ac640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 0x7fb69413f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f416a3d4640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7f416a3d4640 10 monclient: monmap: 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.659 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 -- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 msgr2=0x7fb69413f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 0x7fb69413f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8d8c0302f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 0x7f8d8c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8d8c0332f0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c030710 con 0x7f8d8c0332f0 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c0d6220 con 0x7f8d8c006de0 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8c0cc020 con 0x7f8d8c0302f0 2024-09-16T03:49:57.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d99e8f640 10 monclient(hunting): authenticate will time out at 756.214133s 2024-09-16T03:49:57.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4800caa20 0x7fd48002dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4800caa20 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800d2fd0 con 0x7fd4800caa20 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800c50f0 con 0x7fd480006e30 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fd4800d3150 con 0x7fd480034b90 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48759e640 10 monclient(hunting): authenticate will time out at 756.214129s 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): get_auth_request con 0x7fd480006e30 auth_method 0 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): get_auth_request con 0x7fd480034b90 auth_method 0 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): get_auth_request con 0x7fd4800caa20 auth_method 0 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd48659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7feef2e12640 1 -- 172.21.15.70:0/171762738 wait complete. 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 librados: starting msgr at 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 librados: starting objecter 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 Processor -- start 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe80c023640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe80c023640 10 monclient: monmap: 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 librados: starting msgr at 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 librados: starting objecter 2024-09-16T03:49:57.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 Processor -- start 2024-09-16T03:49:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 0x7f8d8c0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 -- 172.21.15.70:0/1666014029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb694146c10 con 0x7fb6940aaf50 2024-09-16T03:49:57.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 0x7f8d8c0c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34854/0 (socket says 172.21.15.70:34854) 2024-09-16T03:49:57.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 1 -- 172.21.15.70:0/2393844983 learned_addr learned my addr 172.21.15.70:0/2393844983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 10 monclient(hunting): get_auth_request con 0x7f8d8c006de0 auth_method 0 2024-09-16T03:49:57.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d98e8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 -- start start 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 librados: setting wanted keys 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 1 librados: calling monclient init 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f416a3d4640 10 monclient: init 2024-09-16T03:49:57.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a8ae640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 0x7fb694144f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c410 tx=0x7fb69000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 client.?.objecter ms_handle_connect 0x7fb6940aaf50 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-c con 0x7fe64c006b40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690017070 con 0x7fb6940aaf50 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: dump: 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6900035f0 con 0x7fb6940aaf50 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900039e0 con 0x7fb6940aaf50 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb68affd640 10 monclient: dump: 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.672 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69a0ad640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69b0af640 5 monclient: authenticate success, global_id 4360 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69b0af640 10 monclient: _renew_subs 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.551+0000 7fe65932a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c0c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb69b0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 1 -- 172.21.15.70:0/1666014029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb694146ef0 con 0x7fb6940aaf50 2024-09-16T03:49:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-a con 0x7fe64c031980 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd486d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 -- 172.21.15.70:0/385473594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 msgr2=0x7fd48002fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 0x7fd48002fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 -- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4800caa20 msgr2=0x7fd48002dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4800caa20 0x7fd48002dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fd485d9b640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48002efc0 con 0x7fd480034b90 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb6998ac640 10 monclient: get_auth_request con 0x7fb6940aa780 auth_method 0 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb6998ac640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 0x7fb69413f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 10 monclient: _renew_subs 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 1 -- 172.21.15.70:0/1666014029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb694147790 con 0x7fb6940aaf50 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fb69b0af640 1 librados: init done 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 -- start start 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 librados: setting wanted keys 2024-09-16T03:49:57.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 1 librados: calling monclient init 2024-09-16T03:49:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe80c023640 10 monclient: init 2024-09-16T03:49:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe64c0c40f0 0x7fe64c1641c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fb6988aa640 4 set_mon_vals no callback set 2024-09-16T03:49:57.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): picked mon.noname-b con 0x7fe64c0c40f0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d9968e640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 0x7f8d8c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c02fda0 con 0x7fe64c031980 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d9968e640 10 monclient(hunting): get_auth_request con 0x7f8d8c0332f0 auth_method 0 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d9968e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d9968e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7f8d9968e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d98e8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d93fff640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c0302f0 0x7f8d8c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d93fff640 10 monclient(hunting): get_auth_request con 0x7f8d8c0302f0 auth_method 0 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 -- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c0302f0 msgr2=0x7f8d8c0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c0302f0 0x7f8d8c0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 -- 172.21.15.70:0/2393844983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 msgr2=0x7f8d8c0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 0x7f8d8c0c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 -- 172.21.15.70:0/2393844983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c0c5930 con 0x7f8d8c0332f0 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7feef2e12640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c02c4c0 con 0x7fe64c0c40f0 2024-09-16T03:49:57.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c02ff20 con 0x7fe64c006b40 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe65932a640 10 monclient(hunting): authenticate will time out at 756.214728s 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe80c023640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45840/0 (socket says 172.21.15.70:45840) 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 1 -- 172.21.15.70:0/2896495185 learned_addr learned my addr 172.21.15.70:0/2896495185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 10 monclient(hunting): get_auth_request con 0x7fe64c006b40 auth_method 0 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.552+0000 7fe652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7feef2e12640 10 monclient: monmap: 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.683 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe80c023640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:57.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe80c023640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8040ac0f0 0x7fe804132180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe653fff640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe64c0c40f0 0x7fe64c1641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-b con 0x7f41640aaa40 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 0x7f41641c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe64c0c40f0 auth_method 0 2024-09-16T03:49:57.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-a con 0x7f41640af940 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f41641c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4164142f50 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d9968e640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 0x7f8d8c032330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d94065710 tx=0x7f8d9406b270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d917fa640 10 client.?.objecter ms_handle_connect 0x7f8d8c0332f0 2024-09-16T03:49:57.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f8d917fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41640ab640 con 0x7f41640af940 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41640acf70 con 0x7f41640aaa40 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f41640ab030 con 0x7f4164142f50 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f416a3d4640 10 monclient(hunting): authenticate will time out at 756.215828s 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f41637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 0x7f41641c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34874/0 (socket says 172.21.15.70:34874) 2024-09-16T03:49:57.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 1 -- 172.21.15.70:0/1866700965 learned_addr learned my addr 172.21.15.70:0/1866700965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 10 monclient(hunting): get_auth_request con 0x7f41640aaa40 auth_method 0 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7f4163fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 -- 172.21.15.70:0/2896495185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe64c0c40f0 msgr2=0x7fe64c1641c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe64c0c40f0 0x7fe64c1641c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 -- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 msgr2=0x7fe64c0c68c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4168971640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f41641c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c0c68c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4168971640 10 monclient(hunting): get_auth_request con 0x7f4164142f50 auth_method 0 2024-09-16T03:49:57.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f7c2e586640 1 -- 172.21.15.70:0/2650237154 shutdown_connections 2024-09-16T03:49:57.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 librados: starting msgr at 2024-09-16T03:49:57.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4168971640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 librados: starting objecter 2024-09-16T03:49:57.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4168971640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4168971640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-09-16T03:49:57.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 -- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 msgr2=0x7f41641c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f41641c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 -- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 msgr2=0x7f41641c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 Processor -- start 2024-09-16T03:49:57.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 -- start start 2024-09-16T03:49:57.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 librados: setting wanted keys 2024-09-16T03:49:57.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 1 librados: calling monclient init 2024-09-16T03:49:57.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7feef2e12640 10 monclient: init 2024-09-16T03:49:57.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe809dbf640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8040ac0f0 0x7fe804132180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-c con 0x7fe8040ac0f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-a con 0x7fe80413c1c0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe804142700 0x7fe8041344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): picked mon.noname-b con 0x7fe804142700 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8040ab190 con 0x7fe80413c1c0 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe80413cb10 con 0x7fe804142700 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fe8040ab310 con 0x7fe8040ac0f0 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80c023640 10 monclient(hunting): authenticate will time out at 756.216583s 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49196/0 (socket says 172.21.15.70:49196) 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 1 -- 172.21.15.70:0/3935033713 learned_addr learned my addr 172.21.15.70:0/3935033713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): get_auth_request con 0x7fe80413c1c0 auth_method 0 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 0x7f41641c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 monclient: found mon.noname-b 2024-09-16T03:49:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe8095be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f4163fff640 1 -- 172.21.15.70:0/1866700965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41641cb230 con 0x7f41640aaa40 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe809dbf640 10 monclient(hunting): get_auth_request con 0x7fe8040ac0f0 auth_method 0 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe809dbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe809dbf640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe809dbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80a5c0640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe804142700 0x7fe8041344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80a5c0640 10 monclient(hunting): get_auth_request con 0x7fe804142700 auth_method 0 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80a5c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80a5c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe80a5c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fd485d9b640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd488051790 tx=0x7fd488069b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fd476ffd640 10 client.?.objecter ms_handle_connect 0x7fd480034b90 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fd476ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f41637fe640 10 monclient: get_auth_request con 0x7f41640af940 auth_method 0 2024-09-16T03:49:57.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f41637fe640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 0x7f41641c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.553+0000 7fe652ffd640 1 -- 172.21.15.70:0/2896495185 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c0c5070 con 0x7fe64c006b40 2024-09-16T03:49:57.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94069020 con 0x7f8d8c0332f0 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7f8d917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4880b3050 con 0x7fd480034b90 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: dump: 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.703 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4880b4040 con 0x7fd480034b90 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4880adcb0 con 0x7fd480034b90 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd476ffd640 10 monclient: dump: 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 5 monclient: authenticate success, global_id 4289 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 10 monclient: _renew_subs 2024-09-16T03:49:57.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48002f2a0 con 0x7fd480034b90 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 10 monclient: _renew_subs 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48002d150 con 0x7fd480034b90 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fd48759e640 1 librados: init done 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fd48759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fd48759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fd48759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd48002d350 con 0x7fd480034b90 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d917fa640 10 monclient: dump: 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d917fa640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d917fa640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 5 monclient: authenticate success, global_id 4341 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 10 monclient: _renew_subs 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 1 -- 172.21.15.70:0/2393844983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c0c8ef0 con 0x7f8d8c0332f0 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 10 monclient: _renew_subs 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 1 -- 172.21.15.70:0/2393844983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c032870 con 0x7f8d8c0332f0 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f8d99e8f640 1 librados: init done 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9406b3a0 con 0x7f8d8c0332f0 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d7effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94072870 con 0x7f8d8c0332f0 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d7effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d7effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7feef2e12640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7feef2e12640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7feef2e12640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeec073680 0x7feeec071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-a con 0x7feeec073680 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7feef2e12640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45874/0 (socket says 172.21.15.70:45874) 2024-09-16T03:49:57.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 1 -- 172.21.15.70:0/3074204144 learned_addr learned my addr 172.21.15.70:0/3074204144 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-c con 0x7feeec0aaa40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0af940 0x7feeec1d4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): picked mon.noname-b con 0x7feeec0af940 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2611640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0af940 0x7feeec1d4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0ab6b0 con 0x7feeec073680 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0afd40 con 0x7feeec0af940 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7feeec0abcb0 con 0x7feeec0aaa40 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2e12640 10 monclient(hunting): authenticate will time out at 756.218616s 2024-09-16T03:49:57.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): get_auth_request con 0x7feeec0aaa40 auth_method 0 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef160f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2611640 10 monclient(hunting): get_auth_request con 0x7feeec0af940 auth_method 0 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2611640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7feef2611640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 -- 172.21.15.70:0/3074204144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0af940 msgr2=0x7feeec1d4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0af940 0x7feeec1d4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 -- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeec073680 msgr2=0x7feeec071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeec073680 0x7feeec071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec1207e0 con 0x7feeec0aaa40 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7feef160f640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feedc00b3c0 tx=0x7feedc00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fb69001a070 con 0x7fb6940aaf50 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 1 --2- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 0x7fb678070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fb69008a100 con 0x7fb6940aaf50 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 10 client.4360.objecter ms_dispatch 0x7fb694000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 3 client.4360.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 3 client.4360.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fb68affd640 20 client.4360.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fb69a0ad640 1 --2- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 0x7fb678070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fb69a0ad640 10 monclient: get_auth_request con 0x7fb67806dcd0 auth_method 0 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fb69a0ad640 1 --2- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 0x7fb678070190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb68c001560 tx=0x7fb68c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fb68affd640 10 client.4360.objecter ms_handle_connect 0x7fb67806dcd0 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-09-16T03:49:57.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 -- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8040ac0f0 msgr2=0x7fe804132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8040ac0f0 0x7fe804132180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 -- 172.21.15.70:0/3935033713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe804142700 msgr2=0x7fe8041344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe804142700 0x7fe8041344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 monclient: found mon.noname-a 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 -- 172.21.15.70:0/3935033713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe804135df0 con 0x7fe80413c1c0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe8095be640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c4a0 tx=0x7fe80000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 10 client.?.objecter ms_handle_connect 0x7fe80413c1c0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800017070 con 0x7fe80413c1c0 2024-09-16T03:49:57.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe7fa7fc640 10 monclient: dump: 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe7fa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe7fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe7fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe80c023640 5 monclient: authenticate success, global_id 4344 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe80c023640 10 monclient: _renew_subs 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe80c023640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe804134a10 con 0x7fe80413c1c0 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7f4163fff640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f415c035040 tx=0x7f415c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 client.?.objecter ms_handle_connect 0x7f41640aaa40 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c037040 con 0x7f41640aaa40 2024-09-16T03:49:57.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 monclient: dump: 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f4160ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f416a3d4640 5 monclient: authenticate success, global_id 4372 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f416a3d4640 10 monclient: _renew_subs 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f416a3d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f416a3d4640 1 -- 172.21.15.70:0/1866700965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41641cb4a0 con 0x7f41640aaa40 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f416a3d4640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f416a3d4640 10 monclient: _renew_subs 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f416a3d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f416a3d4640 1 -- 172.21.15.70:0/1866700965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f416413ca00 con 0x7f41640aaa40 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f416a3d4640 1 librados: init done 2024-09-16T03:49:57.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe652ffd640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e810 tx=0x7fe63c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 client.?.objecter ms_handle_connect 0x7fe64c006b40 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c017070 con 0x7fe64c006b40 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 monclient: dump: 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.719 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.554+0000 7fe64b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 5 monclient: authenticate success, global_id 4292 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 10 monclient: _renew_subs 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c164700 con 0x7fe64c006b40 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 10 monclient: _renew_subs 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64c164bf0 con 0x7fe64c006b40 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c00c6d0 con 0x7fe64c006b40 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe65932a640 1 librados: init done 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00f040 con 0x7fe64c006b40 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe6517fa640 4 set_mon_vals no callback set 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.555+0000 7fe64b7fe640 10 monclient: dump: 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.721 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f7c2e586640 1 -- 172.21.15.70:0/2650237154 wait complete. 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f7c2e586640 10 monclient: build_initial_monmap 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7f7c2e586640 10 monclient: monmap: 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:57.722 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 librados: starting msgr at 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 librados: starting objecter 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 Processor -- start 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f415c042070 con 0x7f41640aaa40 2024-09-16T03:49:57.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c032810 con 0x7f41640aaa40 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f41627fc640 4 set_mon_vals no callback set 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7fe65932a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fe65932a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fe65932a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.557+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe638005080 con 0x7fe64c006b40 2024-09-16T03:49:57.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fe63c01a070 con 0x7fe64c006b40 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f4160ff9640 10 monclient: dump: 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.725 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d74005080 con 0x7f8d8c0332f0 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.556+0000 7f8d92ffd640 4 set_mon_vals no callback set 2024-09-16T03:49:57.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f415a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe80c023640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 -- start start 2024-09-16T03:49:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 librados: setting wanted keys 2024-09-16T03:49:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 1 librados: calling monclient init 2024-09-16T03:49:57.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f7c2e586640 10 monclient: init 2024-09-16T03:49:57.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7feee27fc640 10 client.?.objecter ms_handle_connect 0x7feeec0aaa40 2024-09-16T03:49:57.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7feee27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe80c023640 10 monclient: _renew_subs 2024-09-16T03:49:57.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe64b7fe640 1 --2- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 0x7fe634070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fe63c08aa10 con 0x7fe64c006b40 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe64b7fe640 10 client.4292.objecter ms_dispatch 0x7fe654000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe64b7fe640 3 client.4292.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f415a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe64b7fe640 3 client.4292.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fb69b0af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fb69b0af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fb69b0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fb69b0af640 1 -- 172.21.15.70:0/1666014029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb694147990 con 0x7fb6940aaf50 2024-09-16T03:49:57.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f415a7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fd477fff640 4 set_mon_vals no callback set 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f8d917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f8d917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7f8d917fa640 10 monclient: dump: 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.731 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41640684d0 con 0x7f41640aaa40 2024-09-16T03:49:57.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fe6527fc640 1 --2- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 0x7fe634070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fe64b7fe640 20 client.4292.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fe6527fc640 10 monclient: get_auth_request con 0x7fe63406dcd0 auth_method 0 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c280b28f0 0x7f7c28142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-b con 0x7f7c280b28f0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-c con 0x7f7c280b30f0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c281e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28141940 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2813e7b0 con 0x7f7c28141940 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2813e630 con 0x7f7c280b28f0 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f7c28144d90 con 0x7f7c280b30f0 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7f7c2e586640 10 monclient(hunting): authenticate will time out at 756.221785s 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45880/0 (socket says 172.21.15.70:45880) 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2378952244 learned_addr learned my addr 172.21.15.70:0/2378952244 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request con 0x7f7c280b30f0 auth_method 0 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:57.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2378952244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c280b28f0 msgr2=0x7f7c28142f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c280b28f0 0x7f7c28142f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 msgr2=0x7f7c281e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c281e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 monclient: found mon.noname-c 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 -- 172.21.15.70:0/2378952244 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c281498b0 con 0x7f7c280b30f0 2024-09-16T03:49:57.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fd4880c1030 con 0x7fd480034b90 2024-09-16T03:49:57.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c1ffff640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c180090b0 tx=0x7f7c180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe6527fc640 1 --2- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 0x7fe634070190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe644000f60 tx=0x7fe6440079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 1 --2- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 0x7fd468070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd48659c640 1 --2- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 0x7fd468070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd48659c640 10 monclient: get_auth_request con 0x7fd46806dcd0 auth_method 0 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fd488127d40 con 0x7fd480034b90 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 10 client.4289.objecter ms_dispatch 0x7fd488000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 3 client.4289.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 3 client.4289.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fd476ffd640 20 client.4289.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f415c0329b0 con 0x7f41640aaa40 2024-09-16T03:49:57.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c280b30f0 2024-09-16T03:49:57.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe64b7fe640 10 client.4292.objecter ms_handle_connect 0x7fe63406dcd0 2024-09-16T03:49:57.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:57.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fd48659c640 1 --2- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 0x7fd468070190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd478009990 tx=0x7fd478009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fd476ffd640 10 client.4289.objecter ms_handle_connect 0x7fd46806dcd0 2024-09-16T03:49:57.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe80c023640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe804136210 con 0x7fe80413c1c0 2024-09-16T03:49:57.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe80c023640 1 librados: init done 2024-09-16T03:49:57.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe800005ce0 con 0x7fe80413c1c0 2024-09-16T03:49:57.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800007270 con 0x7fe80413c1c0 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 10 monclient: dump: 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.740 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.558+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fe80001a070 con 0x7fe80413c1c0 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe7fa7fc640 1 --2- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 0x7fe7e8070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe809dbf640 1 --2- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 0x7fe7e8070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe7fbfff640 4 set_mon_vals no callback set 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.559+0000 7fe809dbf640 10 monclient: get_auth_request con 0x7fe7e806dc80 auth_method 0 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fe809dbf640 1 --2- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 0x7fe7e8070140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009a30 tx=0x7fe7f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.560+0000 7fe7fa7fc640 10 client.4344.objecter ms_handle_connect 0x7fe7e806dc80 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe80c023640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe80c023640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe80c023640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8040b22e0 con 0x7fe80413c1c0 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 1 --2- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 0x7f4144070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f41637fe640 1 --2- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 0x7f4144070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f415c0ab7c0 con 0x7f41640aaa40 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 10 client.4372.objecter ms_dispatch 0x7f4164000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 3 client.4372.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f41637fe640 10 monclient: get_auth_request con 0x7f414406dcd0 auth_method 0 2024-09-16T03:49:57.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.561+0000 7f4160ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f8d94072a10 con 0x7f8d8c0332f0 2024-09-16T03:49:57.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f4160ff9640 10 client.4372.objecter ms_handle_connect 0x7f414406dcd0 2024-09-16T03:49:57.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f41637fe640 1 --2- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 0x7f4144070190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41641cb0f0 tx=0x7f414c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 1 --2- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 0x7f8d78070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d98e8d640 1 --2- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 0x7f8d78070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d98e8d640 10 monclient: get_auth_request con 0x7f8d7806dcd0 auth_method 0 2024-09-16T03:49:57.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f8d94106a30 con 0x7f8d8c0332f0 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 10 client.4341.objecter ms_dispatch 0x7f8d94000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 3 client.4341.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 3 client.4341.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.562+0000 7f8d917fa640 20 client.4341.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f8d917fa640 10 client.4341.objecter ms_handle_connect 0x7f8d7806dcd0 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f8d98e8d640 1 --2- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 0x7f8d78070190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009630 tx=0x7f8d8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd48002d350 con 0x7fd480034b90 2024-09-16T03:49:57.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7fd476ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc011830 con 0x7feeec0aaa40 2024-09-16T03:49:57.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18037050 con 0x7f7c280b30f0 2024-09-16T03:49:57.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 10 monclient: dump: 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.747 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 10 monclient: dump: 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.748 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 10 monclient: _finish_auth 0 2024-09-16T03:49:57.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7feee27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 10 monclient: _check_auth_tickets 2024-09-16T03:49:57.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7f7c257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:57.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.563+0000 7fd476ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 5 monclient: authenticate success, global_id 4304 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 10 monclient: _renew_subs 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeec1415d0 con 0x7feeec0aaa40 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 5 monclient: authenticate success, global_id 4307 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 10 monclient: _renew_subs 2024-09-16T03:49:57.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c28145070 con 0x7f7c280b30f0 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 10 monclient: _renew_subs 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feeec11f8c0 con 0x7feeec0aaa40 2024-09-16T03:49:57.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7feef2e12640 1 librados: init done 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 10 monclient: _renew_subs 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c281e0fe0 con 0x7f7c280b30f0 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.564+0000 7f7c2e586640 1 librados: init done 2024-09-16T03:49:57.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7f7c2e586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb694147990 con 0x7fb6940aaf50 2024-09-16T03:49:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fb68affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fb68affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7f7c2e586640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7f7c2e586640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28144120 con 0x7f7c280b30f0 2024-09-16T03:49:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7feef2e12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7feef2e12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7feef2e12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeec06d870 con 0x7feeec0aaa40 2024-09-16T03:49:57.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fe800056710 con 0x7fe80413c1c0 2024-09-16T03:49:57.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fe7fa7fc640 10 client.4344.objecter ms_dispatch 0x7fe804000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fe7fa7fc640 3 client.4344.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.565+0000 7fe7fa7fc640 3 client.4344.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.566+0000 7fe7fa7fc640 20 client.4344.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.566+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8d940a4080 con 0x7f8d8c0332f0 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.566+0000 7f8d917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.566+0000 7f8d917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feedc016070 con 0x7feeec0aaa40 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc004220 con 0x7feeec0aaa40 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7feee27fc640 10 monclient: dump: 2024-09-16T03:49:57.757 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.758 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18043070 con 0x7f7c280b30f0 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18030c10 con 0x7f7c280b30f0 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c26ffd640 4 set_mon_vals no callback set 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.568+0000 7f7c257fa640 10 monclient: dump: 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:57.760 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:57.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.569+0000 7feee3fff640 4 set_mon_vals no callback set 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.569+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe63c059770 con 0x7fe64c006b40 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.569+0000 7fe64b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.569+0000 7fe64b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.570+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f41641cb4a0 con 0x7f41640aaa40 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.570+0000 7f4160ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.570+0000 7f4160ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.570+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7feedc0119d0 con 0x7feeec0aaa40 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 1 --2- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 0x7feed00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7feedc08af40 con 0x7feeec0aaa40 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 10 client.4304.objecter ms_dispatch 0x7feeec000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 3 client.4304.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 3 client.4304.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.572+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe800058d70 con 0x7fe80413c1c0 2024-09-16T03:49:57.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.572+0000 7fe7fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.572+0000 7fe7fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.571+0000 7feee27fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.572+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f7c18048460 con 0x7f7c280b30f0 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.572+0000 7f7c257fa640 1 --2- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 0x7f7c080700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f7c180ac850 con 0x7f7c280b30f0 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7f7c257fa640 10 client.4307.objecter ms_dispatch 0x7f7c28000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7f7c257fa640 3 client.4307.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7f7c257fa640 3 client.4307.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:57.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7f7c257fa640 20 client.4307.objecter dump_active .. 0 homeless 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7feef1e10640 1 --2- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 0x7feed00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.573+0000 7feef1e10640 10 monclient: get_auth_request con 0x7feed006dc00 auth_method 0 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.574+0000 7feef1e10640 1 --2- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 0x7feed00700c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feee8037000 tx=0x7feee8032000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.574+0000 7feee27fc640 10 client.4304.objecter ms_handle_connect 0x7feed006dc00 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.574+0000 7f7c27fff640 1 --2- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 0x7f7c080700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.574+0000 7f7c27fff640 10 monclient: get_auth_request con 0x7f7c0806dc00 auth_method 0 2024-09-16T03:49:57.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.575+0000 7f7c257fa640 10 client.4307.objecter ms_handle_connect 0x7f7c0806dc00 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.575+0000 7f7c27fff640 1 --2- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 0x7f7c080700c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c100096b0 tx=0x7f7c10006c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.579+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7feedc020020 con 0x7feeec0aaa40 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.579+0000 7feee27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.579+0000 7feee27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.588+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7c1807b590 con 0x7f7c280b30f0 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.588+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:57.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.588+0000 7f7c257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:57.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.961+0000 7fd48759e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:57.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.961+0000 7fd48759e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:57.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.961+0000 7fd48759e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.961+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd48002e0f0 con 0x7fd480034b90 2024-09-16T03:49:57.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.962+0000 7fd476ffd640 1 -- 172.21.15.70:0/385473594 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd48002e0f0 con 0x7fd480034b90 2024-09-16T03:49:57.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.962+0000 7fd476ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:57.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.962+0000 7fd476ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.962+0000 7f8d7effd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:57.966 INFO:teuthology.orchestra.run.smithi070.stdout:64424509443 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d7effd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d7effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8d74005640 con 0x7f8d8c0332f0 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d917fa640 1 -- 172.21.15.70:0/2393844983 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d940d5820 con 0x7f8d8c0332f0 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.963+0000 7f8d917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.967 INFO:teuthology.orchestra.run.smithi070.stdout:60129542147 2024-09-16T03:49:57.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.964+0000 7fb69b0af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:57.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.964+0000 7fb69b0af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:57.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.964+0000 7fb69b0af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.964+0000 7fb69b0af640 1 -- 172.21.15.70:0/1666014029 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb6940ae4d0 con 0x7fb6940aaf50 2024-09-16T03:49:57.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.965+0000 7fb68affd640 1 -- 172.21.15.70:0/1666014029 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb6900588a0 con 0x7fb6940aaf50 2024-09-16T03:49:57.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.965+0000 7fb68affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:57.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.965+0000 7fb68affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.969 INFO:teuthology.orchestra.run.smithi070.stdout:60129542147 2024-09-16T03:49:57.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.965+0000 7fd48759e640 10 librados: watch_flush enter 2024-09-16T03:49:57.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 10 librados: watch_flush exit 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 20 client.4289.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 10 client.4289.objecter successfully canceled tick 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 msgr2=0x7fd468070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 0x7fd468070190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd478009990 tx=0x7fd478009110 comp rx=0 tx=0).stop 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 10 monclient: shutdown 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 msgr2=0x7fd48002d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.966+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd488051790 tx=0x7fd488069b20 comp rx=0 tx=0).stop 2024-09-16T03:49:57.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 shutdown_connections 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> v2:172.21.15.70:6804/198622594 conn(0x7fd46806dcd0 0x7fd468070190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4800caa20 0x7fd48002dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd480034b90 0x7fd48002d670 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7f8d7effd640 10 librados: watch_flush enter 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7f8d7effd640 10 librados: watch_flush exit 2024-09-16T03:49:57.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 --2- 172.21.15.70:0/385473594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd480006e30 0x7fd48002fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 >> 172.21.15.70:0/385473594 conn(0x7fd480005d60 msgr2=0x7fd480037770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7f8d7effd640 20 client.4341.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 shutdown_connections 2024-09-16T03:49:57.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 -- 172.21.15.70:0/385473594 wait complete. 2024-09-16T03:49:57.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7f8d7effd640 10 client.4341.objecter successfully canceled tick 2024-09-16T03:49:57.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.967+0000 7fd48759e640 1 librados: shutdown 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 msgr2=0x7f8d78070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 0x7f8d78070190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009630 tx=0x7f8d8002e000 comp rx=0 tx=0).stop 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 10 monclient: shutdown 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 msgr2=0x7f8d8c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 0x7f8d8c032330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d94065710 tx=0x7f8d9406b270 comp rx=0 tx=0).stop 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 shutdown_connections 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> v2:172.21.15.70:6804/198622594 conn(0x7f8d7806dcd0 0x7f8d78070190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c0332f0 0x7f8d8c032330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7fe65932a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:57.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7fe65932a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:57.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7fe65932a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe638005370 con 0x7fe64c006b40 2024-09-16T03:49:57.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8d8c0302f0 0x7f8d8c0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 --2- 172.21.15.70:0/2393844983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d8c006de0 0x7f8d8c0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.968+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 >> 172.21.15.70:0/2393844983 conn(0x7f8d8c005d60 msgr2=0x7f8d8c0c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 shutdown_connections 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7f8d7effd640 1 -- 172.21.15.70:0/2393844983 wait complete. 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7f8d7effd640 1 librados: shutdown 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 10 librados: watch_flush enter 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 10 librados: watch_flush exit 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 20 client.4360.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 10 client.4360.objecter successfully canceled tick 2024-09-16T03:49:57.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fe64b7fe640 1 -- 172.21.15.70:0/2896495185 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe63c059070 con 0x7fe64c006b40 2024-09-16T03:49:57.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fe64b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:57.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fe64b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.979 INFO:teuthology.orchestra.run.smithi070.stdout:64424509444 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 msgr2=0x7fb678070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 0x7fb678070190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb68c001560 tx=0x7fb68c009040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.969+0000 7fb688ff9640 10 monclient: shutdown 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 msgr2=0x7fb694144f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 0x7fb694144f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c410 tx=0x7fb69000c8d0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 shutdown_connections 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> v2:172.21.15.70:6804/198622594 conn(0x7fb67806dcd0 0x7fb678070190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6940aaf50 0x7fb694144f50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6940aa780 0x7fb69413f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 --2- 172.21.15.70:0/1666014029 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fb694072e80 0x7fb694147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 >> 172.21.15.70:0/1666014029 conn(0x7fb6940b46e0 msgr2=0x7fb6941403b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 shutdown_connections 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 -- 172.21.15.70:0/1666014029 wait complete. 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.970+0000 7fb688ff9640 1 librados: shutdown 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 10 librados: watch_flush enter 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 10 librados: watch_flush exit 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 20 client.4292.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 10 client.4292.objecter successfully canceled tick 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe80c023640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 msgr2=0x7fe634070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 0x7fe634070190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe644000f60 tx=0x7fe6440079b0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 10 monclient: shutdown 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 msgr2=0x7fe64c0c3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e810 tx=0x7fe63c00ecd0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.973+0000 7fe80c023640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:57.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe80c023640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:57.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe8040b4820 con 0x7fe80413c1c0 2024-09-16T03:49:57.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 shutdown_connections 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> v2:172.21.15.70:6804/198622594 conn(0x7fe63406dcd0 0x7fe634070190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe64c0c40f0 0x7fe64c1641c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe64c031980 0x7fe64c0c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 --2- 172.21.15.70:0/2896495185 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe64c006b40 0x7fe64c0c3650 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 >> 172.21.15.70:0/2896495185 conn(0x7fe64c005c30 msgr2=0x7fe64c0c0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 shutdown_connections 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 -- 172.21.15.70:0/2896495185 wait complete. 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe7fa7fc640 1 -- 172.21.15.70:0/3935033713 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe800058670 con 0x7fe80413c1c0 2024-09-16T03:49:57.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe7fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:57.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe7fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7f415a7fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:57.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7fe65932a640 1 librados: shutdown 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7f415a7fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7f415a7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.974+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f41640b2f00 con 0x7f41640aaa40 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stdout:60129542147 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.975+0000 7f4160ff9640 1 -- 172.21.15.70:0/1866700965 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f415c07a6d0 con 0x7f41640aaa40 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.975+0000 7f4160ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:57.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.975+0000 7f4160ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stdout:55834574852 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 10 librados: watch_flush enter 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 10 librados: watch_flush exit 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 20 client.4344.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 10 client.4344.objecter successfully canceled tick 2024-09-16T03:49:57.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 msgr2=0x7fe7e8070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 0x7fe7e8070140 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009a30 tx=0x7fe7f4008040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 10 monclient: shutdown 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 msgr2=0x7fe8041378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.978+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c4a0 tx=0x7fe80000c960 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 shutdown_connections 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> v2:172.21.15.70:6804/198622594 conn(0x7fe7e806dc80 0x7fe7e8070140 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009a30 tx=0x7fe7f4008040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe804142700 0x7fe8041344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe80413c1c0 0x7fe8041378a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 --2- 172.21.15.70:0/3935033713 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fe8040ac0f0 0x7fe804132180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 >> 172.21.15.70:0/3935033713 conn(0x7fe8040b1650 msgr2=0x7fe8040b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 10 librados: watch_flush enter 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 10 librados: watch_flush exit 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 20 client.4372.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 10 client.4372.objecter successfully canceled tick 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 msgr2=0x7f4144070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 0x7f4144070190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41641cb0f0 tx=0x7f414c009040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 10 monclient: shutdown 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 msgr2=0x7f41641ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f415c035040 tx=0x7f415c0046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 shutdown_connections 2024-09-16T03:49:57.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> v2:172.21.15.70:6804/198622594 conn(0x7f414406dcd0 0x7f4144070190 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41641cb0f0 tx=0x7f414c009040 comp rx=0 tx=0).stop 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4164142f50 0x7f41641c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41640af940 0x7f41641c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 --2- 172.21.15.70:0/1866700965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41640aaa40 0x7f41641ce190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.979+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 >> 172.21.15.70:0/1866700965 conn(0x7f4164068020 msgr2=0x7f4164072f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 shutdown_connections 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7fe80c023640 1 -- 172.21.15.70:0/3935033713 wait complete. 2024-09-16T03:49:57.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7fe80c023640 1 librados: shutdown 2024-09-16T03:49:57.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7feef2e12640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:57.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7feef2e12640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:57.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7feef2e12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7feeec0b3e90 con 0x7feeec0aaa40 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 shutdown_connections 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.980+0000 7f415a7fc640 1 -- 172.21.15.70:0/1866700965 wait complete. 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.981+0000 7feee27fc640 1 -- 172.21.15.70:0/3074204144 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feedc018310 con 0x7feeec0aaa40 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.981+0000 7feee27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.981+0000 7feee27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.981+0000 7f415a7fc640 1 librados: shutdown 2024-09-16T03:49:57.994 INFO:teuthology.orchestra.run.smithi070.stdout:60129542147 2024-09-16T03:49:57.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7f7c2e586640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:57.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7feef2e12640 10 librados: watch_flush enter 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7f7c2e586640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7f7c2e586640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f7c28143dc0 con 0x7f7c280b30f0 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7feef2e12640 10 librados: watch_flush exit 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7feef2e12640 20 client.4304.objecter shutdown clearing up homeless session... 2024-09-16T03:49:57.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.986+0000 7feef2e12640 10 client.4304.objecter successfully canceled tick 2024-09-16T03:49:57.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 msgr2=0x7feed00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 0x7feed00700c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feee8037000 tx=0x7feee8032000 comp rx=0 tx=0).stop 2024-09-16T03:49:57.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 10 monclient: shutdown 2024-09-16T03:49:57.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:57.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 msgr2=0x7feeec11e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:57.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feedc00b3c0 tx=0x7feedc00b880 comp rx=0 tx=0).stop 2024-09-16T03:49:57.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.4 2024-09-16T03:49:57.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 shutdown_connections 2024-09-16T03:49:57.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> v2:172.21.15.70:6804/198622594 conn(0x7feed006dc00 0x7feed00700c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feeec0af940 0x7feeec1d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feeec0aaa40 0x7feeec11e460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 --2- 172.21.15.70:0/3074204144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeec073680 0x7feeec071a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 >> 172.21.15.70:0/3074204144 conn(0x7feeec068470 msgr2=0x7feeec141f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7f7c257fa640 1 -- 172.21.15.70:0/2378952244 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7c1807ae90 con 0x7f7c280b30f0 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.987+0000 7f7c257fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:57.999 INFO:teuthology.orchestra.run.smithi070.stdout:60129542146 2024-09-16T03:49:58.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.988+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 shutdown_connections 2024-09-16T03:49:58.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.989+0000 7feef2e12640 1 -- 172.21.15.70:0/3074204144 wait complete. 2024-09-16T03:49:58.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.989+0000 7feef2e12640 1 librados: shutdown 2024-09-16T03:49:58.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.3 2024-09-16T03:49:58.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 10 librados: watch_flush enter 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 10 librados: watch_flush exit 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 20 client.4307.objecter shutdown clearing up homeless session... 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 10 client.4307.objecter successfully canceled tick 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 msgr2=0x7f7c080700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 0x7f7c080700c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c100096b0 tx=0x7f7c10006c20 comp rx=0 tx=0).stop 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 10 monclient: shutdown 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 msgr2=0x7f7c281e0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.991+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c180090b0 tx=0x7f7c180046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 shutdown_connections 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> v2:172.21.15.70:6804/198622594 conn(0x7f7c0806dc00 0x7f7c080700c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c28141940 0x7f7c281e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:58.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7c280b30f0 0x7f7c281e0560 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 --2- 172.21.15.70:0/2378952244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c280b28f0 0x7f7c28142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 >> 172.21.15.70:0/2378952244 conn(0x7f7c280b3770 msgr2=0x7f7c28070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 shutdown_connections 2024-09-16T03:49:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 -- 172.21.15.70:0/2378952244 wait complete. 2024-09-16T03:49:58.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:57.992+0000 7f7c2e586640 1 librados: shutdown 2024-09-16T03:49:58.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.2 2024-09-16T03:49:58.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.6 2024-09-16T03:49:58.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-09-16T03:49:58.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.1 2024-09-16T03:49:58.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542147 for osd.5 2024-09-16T03:49:58.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542146 for osd.7 2024-09-16T03:49:58.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:49:59.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:49:59.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:49:59.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:49:59.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:49:59.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:49:59.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:49:59.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:49:59.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.143+0000 7fbbece53640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.143+0000 7fbbece53640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.144+0000 7fbbece53640 10 monclient: monmap: 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.145518+0000 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.145518+0000 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.147 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.148 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.148 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.148 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.148 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.145+0000 7fbbece53640 1 Processor -- start 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.145+0000 7fbbece53640 1 -- start start 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.145+0000 7fbbece53640 10 monclient: init 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbbe80b2f60 0x7fbbe80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe80b2f60 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbbe814af80 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe8144d90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe806e150 con 0x7fbbe814af80 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbbe80b3340 con 0x7fbbe80b2f60 2024-09-16T03:49:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe814aa60 con 0x7fbbe8144d90 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbece53640 10 monclient(hunting): authenticate will time out at 757.808400s 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe814b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45894/0 (socket says 172.21.15.70:45894) 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 1 -- 172.21.15.70:0/1128382697 learned_addr learned my addr 172.21.15.70:0/1128382697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbe8144d90 auth_method 0 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.146+0000 7fbbe659c640 1 --2- 172.21.15.70:0/1128382697 >> v2:172.21.15.70:3300/0 conn(0x7fbbe80b2f60 0x7fbbe80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): get_auth_request con 0x7fbbe80b2f60 auth_method 0 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 1 -- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 msgr2=0x7fbbe814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 1 --2- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe814b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 1 -- 172.21.15.70:0/1128382697 >> v1:172.21.15.163:6789/0 conn(0x7fbbe814af80 legacy=0x7fbbe8144680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 1 monclient: found mon.noname-a 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.147+0000 7fbbe659c640 1 -- 172.21.15.70:0/1128382697 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe80b4f60 con 0x7fbbe80b2f60 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.148+0000 7fbbe659c640 1 --2- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe80b2f60 0x7fbbe80b4450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80b6910 tx=0x7fbbdc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cd473076c73045 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 1 -- 172.21.15.70:0/1128382697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc037070 con 0x7fbbe80b2f60 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: dump: 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.162 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbece53640 5 monclient: authenticate success, global_id 4347 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbece53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 1 -- 172.21.15.70:0/1128382697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbdc02ae50 con 0x7fbbe80b2f60 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbe559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbece53640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.149+0000 7fbbece53640 4 set_mon_vals no callback set 2024-09-16T03:49:59.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.150+0000 7fbbece53640 10 monclient: shutdown 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.150+0000 7fbbece53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.150+0000 7fbbece53640 1 -- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe80b2f60 msgr2=0x7fbbe80b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.150+0000 7fbbece53640 1 --2- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe80b2f60 0x7fbbe80b4450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80b6910 tx=0x7fbbdc02a140 comp rx=0 tx=0).stop 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.151+0000 7fbbece53640 1 -- 172.21.15.70:0/1128382697 shutdown_connections 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.151+0000 7fbbece53640 1 --2- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.151+0000 7fbbece53640 1 --2- 172.21.15.70:0/1128382697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe80b2f60 0x7fbbe80b4450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.151+0000 7fbbece53640 1 -- 172.21.15.70:0/1128382697 >> 172.21.15.70:0/1128382697 conn(0x7fbbe80679c0 msgr2=0x7fbbe8067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.153+0000 7fbbece53640 1 -- 172.21.15.70:0/1128382697 shutdown_connections 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.153+0000 7fbbece53640 1 -- 172.21.15.70:0/1128382697 wait complete. 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.154+0000 7fbbece53640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.154+0000 7fbbece53640 10 monclient: monmap: 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.164 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.154+0000 7fbbece53640 1 librados: starting msgr at 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.154+0000 7fbbece53640 1 librados: starting objecter 2024-09-16T03:49:59.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.154+0000 7fbbece53640 1 Processor -- start 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.156+0000 7fbbece53640 1 -- start start 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.156+0000 7fbbece53640 1 librados: setting wanted keys 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.156+0000 7fbbece53640 1 librados: calling monclient init 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.156+0000 7fbbece53640 10 monclient: init 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe80b2f60 0x7fbbe8137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe80b2f60 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe8144d90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 0x7fbbe81468c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe814af80 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe80aa660 con 0x7fbbe814af80 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe80aa4e0 con 0x7fbbe80b2f60 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe80aa7e0 con 0x7fbbe8144d90 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbece53640 10 monclient(hunting): authenticate will time out at 757.819550s 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45906/0 (socket says 172.21.15.70:45906) 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 1 -- 172.21.15.70:0/78916369 learned_addr learned my addr 172.21.15.70:0/78916369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 10 monclient(hunting): get_auth_request con 0x7fbbe8144d90 auth_method 0 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.157+0000 7fbbe5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe6d9d640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 0x7fbbe81468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe6d9d640 10 monclient(hunting): get_auth_request con 0x7fbbe814af80 auth_method 0 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 1 -- 172.21.15.70:0/78916369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe80b2f60 msgr2=0x7fbbe8137040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe80b2f60 0x7fbbe8137040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.158+0000 7fbbe5d9b640 1 -- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 msgr2=0x7fbbe81468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 0x7fbbe81468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.159+0000 7fbbe5d9b640 1 -- 172.21.15.70:0/78916369 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe8138280 con 0x7fbbe8144d90 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbe6d9d640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 0x7fbbe81468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbe5d9b640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b3e0 tx=0x7fbbd800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 client.?.objecter ms_handle_connect 0x7fbbe8144d90 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8015020 con 0x7fbbe8144d90 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 monclient: dump: 2024-09-16T03:49:59.177 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 5 monclient: authenticate success, global_id 4313 2024-09-16T03:49:59.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 10 monclient: _renew_subs 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe8138560 con 0x7fbbe8144d90 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 10 monclient: _renew_subs 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe8138b20 con 0x7fbbe8144d90 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.160+0000 7fbbece53640 1 librados: init done 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.164+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd8011850 con 0x7fbbe8144d90 2024-09-16T03:49:59.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.164+0000 7fbbd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.164+0000 7fbbd7fff640 4 set_mon_vals no callback set 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.165+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8011c40 con 0x7fbbe8144d90 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbece53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbd6ffd640 10 monclient: dump: 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.180 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fbbd8018020 con 0x7fbbe8144d90 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.166+0000 7fbbece53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.167+0000 7fbbece53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.167+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc005080 con 0x7fbbe8144d90 2024-09-16T03:49:59.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.176+0000 7fbbd6ffd640 1 --2- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 0x7fbbc80701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fbbd808ad90 con 0x7fbbe8144d90 2024-09-16T03:49:59.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 10 client.4313.objecter ms_dispatch 0x7fbbe8000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 3 client.4313.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbe659c640 1 --2- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 0x7fbbc80701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbe659c640 10 monclient: get_auth_request con 0x7fbbc806dd00 auth_method 0 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbbd801f020 con 0x7fbbe8144d90 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.177+0000 7fbbd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.178+0000 7fbbe659c640 1 --2- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 0x7fbbc80701c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc034000 tx=0x7fbbdc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.178+0000 7fbbd6ffd640 10 client.4313.objecter ms_handle_connect 0x7fbbc806dd00 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7f779ab11640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7f779ab11640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7f779ab11640 10 monclient: monmap: 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.208728+0000 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.208728+0000 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.210 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7fd233d54640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7fd233d54640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.207+0000 7fd233d54640 10 monclient: monmap: 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.208550+0000 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.208550+0000 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.213 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f6122b99640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7fd233d54640 1 Processor -- start 2024-09-16T03:49:59.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f9773fff640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f9773fff640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f9773fff640 10 monclient: monmap: 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.209905+0000 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.209905+0000 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f779ab11640 1 Processor -- start 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f779ab11640 1 -- start start 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f779ab11640 10 monclient: init 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T03:49:59.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f6122b99640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7f6122b99640 10 monclient: monmap: 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.209806+0000 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.209806+0000 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.216 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 0x7f77940b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-b con 0x7f77940b5470 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f779413cbd0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fcdd290b640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fcdd290b640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fcdd290b640 10 monclient: monmap: 2024-09-16T03:49:59.217 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.210843+0000 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.210843+0000 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.218 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f6122b99640 1 Processor -- start 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f77990ae640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f779413cff0 0x7f7794143750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33734/0 (socket says 172.21.15.70:33734) 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f779413cff0 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77940ac040 con 0x7f779413cff0 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f779413d420 con 0x7f779413cbd0 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f77940ac2b0 con 0x7f77940b5470 2024-09-16T03:49:59.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f779ab11640 10 monclient(hunting): authenticate will time out at 757.871541s 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f77990ae640 1 -- 172.21.15.70:0/3147468952 learned_addr learned my addr 172.21.15.70:0/3147468952 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7fd233d54640 1 -- start start 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.208+0000 7fd233d54640 10 monclient: init 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd22c0aaa40 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd22c0afb10 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd22c0adaa0 0x7fd22c0adf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c0adaa0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22c0aba90 con 0x7fd22c0aaa40 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22c0ac9b0 con 0x7fd22c0afb10 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fd22c0736d0 con 0x7fd22c0adaa0 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd233d54640 10 monclient(hunting): authenticate will time out at 757.871670s 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f9773fff640 1 Processor -- start 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f9773fff640 1 -- start start 2024-09-16T03:49:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7f9773fff640 10 monclient: init 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.209+0000 7fd2322f1640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd22c0adaa0 0x7fd22c0adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2312ef640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd22c0afb10 0x7fd22c0ad390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:36088/0 (socket says 172.21.15.70:36088) 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fd22c0adaa0 0x7fd22c0adf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34918/0 (socket says 172.21.15.70:34918) 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2312ef640 1 -- 172.21.15.70:0/1486305356 learned_addr learned my addr 172.21.15.70:0/1486305356 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f6122b99640 1 -- start start 2024-09-16T03:49:59.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f6122b99640 10 monclient: init 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): get_auth_request con 0x7fd22c0adaa0 auth_method 0 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611726180 0 0) 0x7f77940ac040 con 0x7f779413cff0 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _negotiate 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): my global_id is 4353 2024-09-16T03:49:59.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7784003520 con 0x7f779413cff0 2024-09-16T03:49:59.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fcdd290b640 1 Processor -- start 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589706369 0 0) 0x7f7780001110 con 0x7f779413cbd0 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _negotiate 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): my global_id is 4387 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77940ac040 con 0x7f779413cbd0 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 1 -- 172.21.15.70:0/1486305356 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2171979408 0 0) 0x7fd22c0ac9b0 con 0x7fd22c0afb10 2024-09-16T03:49:59.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 10 monclient(hunting): _negotiate 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 635489054 0 0) 0x7f7784003520 con 0x7f779413cff0 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 msgr2=0x7f77940b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 0x7f77940b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 10 monclient(hunting): my global_id is 4384 2024-09-16T03:49:59.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd230aee640 1 -- 172.21.15.70:0/1486305356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd218003520 con 0x7fd22c0afb10 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 >> v1:172.21.15.163:6789/0 conn(0x7f779413cbd0 legacy=0x7f7794143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 1 -- 172.21.15.70:0/1486305356 >> v1:172.21.15.163:6789/0 conn(0x7fd22c0afb10 legacy=0x7fd22c0ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 1 -- 172.21.15.70:0/1486305356 >> v1:172.21.15.70:6789/0 conn(0x7fd22c0aaa40 legacy=0x7fd22c0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7fd2322f1640 1 -- 172.21.15.70:0/1486305356 --> v2:172.21.15.163:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22c0acc90 con 0x7fd22c0adaa0 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T03:49:59.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 0x7f976c0cee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-c con 0x7f976c006ee0 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 0x7f976c034890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-b con 0x7f976c0cf3d0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f976c0091d0 0x7f976c034dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-a con 0x7f976c0091d0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f976c0072c0 con 0x7f976c0091d0 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f976c008cf0 con 0x7f976c006ee0 2024-09-16T03:49:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f976c032c30 con 0x7f976c0cf3d0 2024-09-16T03:49:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f9773fff640 10 monclient(hunting): authenticate will time out at 757.872890s 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f976c0091d0 0x7f976c034dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f976c0091d0 0x7f976c034dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49260/0 (socket says 172.21.15.70:49260) 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 1 -- 172.21.15.70:0/2136361296 learned_addr learned my addr 172.21.15.70:0/2136361296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): get_auth_request con 0x7f976c0091d0 auth_method 0 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 0x7f976c034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): get_auth_request con 0x7f976c0cf3d0 auth_method 0 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77940aaef0 con 0x7f779413cff0 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f778c0019f0 con 0x7f779413cff0 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: dump: 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 >> v1:172.21.15.70:6789/0 conn(0x7f779413cff0 legacy=0x7f7794143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:59.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7784003b70 0x7f7784003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): picked mon.b con 0x7f7784003b70 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7784004800 0x7f7784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): picked mon.a con 0x7f7784004800 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 0x7f778400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): picked mon.c con 0x7f77940b5470 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77940ac2b0 con 0x7f7784004800 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7784005520 con 0x7f7784003b70 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f778400bf00 con 0x7f77940b5470 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7784004800 0x7f7784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): get_auth_request con 0x7f7784004800 auth_method 0 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77990ae640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 0x7f778400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77990ae640 10 monclient(hunting): get_auth_request con 0x7f77940b5470 auth_method 0 2024-09-16T03:49:59.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77990ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77990ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f77990ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 293 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 -- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 msgr2=0x7f778400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77940b5470 0x7f778400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 -- 172.21.15.70:0/3147468952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7784003b70 msgr2=0x7f7784003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7784003b70 0x7f7784003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 monclient: found mon.a 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 1 -- 172.21.15.70:0/3147468952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7784003520 con 0x7f7784004800 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f7793fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f779ab11640 5 monclient: authenticate success, global_id 4353 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f779ab11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f611c14aa70 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 0x7f611c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-a con 0x7f611c09bbe0 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6121b97640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f611c14aa70 0x7f611c0b48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:36098/0 (socket says 172.21.15.70:36098) 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6121b97640 1 -- 172.21.15.70:0/3775732971 learned_addr learned my addr 172.21.15.70:0/3775732971 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7efd6dc3c640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7efd6dc3c640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7efd6dc3c640 10 monclient: monmap: 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.213170+0000 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.213170+0000 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7fd2322f1640 1 --2- 172.21.15.70:0/1486305356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0adaa0 0x7fd22c0adf50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd22400aea0 tx=0x7fd22402c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bce2a87495d68e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 1 --2- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c151b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7fcdd290b640 1 -- start start 2024-09-16T03:49:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7fcdd290b640 10 monclient: init 2024-09-16T03:49:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-b con 0x7f611c09e060 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.211+0000 7f97727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f9772ffd640 1 --2- 172.21.15.70:0/2136361296 >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 0x7f976c0cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2024-09-16T03:49:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 -- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 msgr2=0x7f976c034890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f7793fff640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7784004800 0x7f7784004c80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7780001ac0 tx=0x7f7784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a38d231c820ac4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 1 -- 172.21.15.70:0/1486305356 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd224008640 con 0x7fd22c0adaa0 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: dump: 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.245 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 1 -- 172.21.15.70:0/1486305356 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd224037070 con 0x7fd22c0adaa0 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 1 -- 172.21.15.70:0/1486305356 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd224008a20 con 0x7fd22c0adaa0 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:59.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd230aee640 10 monclient: dump: 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd233d54640 5 monclient: authenticate success, global_id 4381 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd233d54640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fd233d54640 4 set_mon_vals no callback set 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 0x7f976c034890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 -- 172.21.15.70:0/2136361296 >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 msgr2=0x7f976c0cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 --2- 172.21.15.70:0/2136361296 >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 0x7f976c0cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97737fe640 1 -- 172.21.15.70:0/2136361296 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c032f10 con 0x7f976c0091d0 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f9772ffd640 10 monclient: get_auth_request con 0x7f976c006ee0 auth_method 0 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f9772ffd640 1 --2- 172.21.15.70:0/2136361296 >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 0x7f976c0cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f97727fc640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 0x7f976c034890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780001690 con 0x7f7784004800 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 10 monclient: dump: 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.249 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f4fbb3c5640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f4fbb3c5640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.210+0000 7f4fbb3c5640 10 monclient: monmap: 2024-09-16T03:49:59.250 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.212140+0000 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.212140+0000 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.251 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7efd6dc3c640 1 Processor -- start 2024-09-16T03:49:59.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f779ab11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdc4006c00 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 0x7fcdc40c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc4030730 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdc402c070 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7fcdd290b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc40c4ed0 con 0x7fcdc402c070 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1909640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcdc4006c00 0x7fcdc402f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:36108/0 (socket says 172.21.15.70:36108) 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1909640 1 -- 172.21.15.70:0/250991534 learned_addr learned my addr 172.21.15.70:0/250991534 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc4031ee0 con 0x7fcdc4006c00 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd290b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc4006fe0 con 0x7fcdc4030730 2024-09-16T03:49:59.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd290b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd290b640 10 monclient(hunting): authenticate will time out at 757.875164s 2024-09-16T03:49:59.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 10 monclient: shutdown 2024-09-16T03:49:59.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 -- 172.21.15.70:0/1486305356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0adaa0 msgr2=0x7fd22c0adf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 --2- 172.21.15.70:0/1486305356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0adaa0 0x7fd22c0adf50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd22400aea0 tx=0x7fd22402c970 comp rx=0 tx=0).stop 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 -- 172.21.15.70:0/1486305356 shutdown_connections 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 --2- 172.21.15.70:0/1486305356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0adaa0 0x7fd22c0adf50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 -- 172.21.15.70:0/1486305356 >> 172.21.15.70:0/1486305356 conn(0x7fd22c068020 msgr2=0x7fd22c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 1 -- 172.21.15.70:0/250991534 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916440770 0 0) 0x7fcdc4031ee0 con 0x7fcdc4006c00 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 10 monclient(hunting): _negotiate 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 10 monclient(hunting): my global_id is 4390 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd0907640 1 -- 172.21.15.70:0/250991534 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdb8003520 con 0x7fcdc4006c00 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77800050d0 con 0x7f7784004800 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f77937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f779ab11640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f779ab11640 4 set_mon_vals no callback set 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f77937fe640 1 -- 172.21.15.70:0/3147468952 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780001690 con 0x7f7784004800 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f77937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f77937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f77937fe640 10 monclient: dump: 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 10 monclient: shutdown 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 1 -- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7784004800 msgr2=0x7f7784004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 1 --2- 172.21.15.70:0/3147468952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7784004800 0x7f7784004c80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7780001ac0 tx=0x7f7784005830 comp rx=0 tx=0).stop 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f77990ae640 1 -- 172.21.15.70:0/3147468952 reap_dead start 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 1 -- 172.21.15.70:0/3147468952 shutdown_connections 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f779ab11640 1 -- 172.21.15.70:0/3147468952 >> 172.21.15.70:0/3147468952 conn(0x7f77940679c0 msgr2=0x7f7794067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 -- 172.21.15.70:0/1486305356 shutdown_connections 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fd233d54640 1 -- 172.21.15.70:0/1486305356 wait complete. 2024-09-16T03:49:59.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f97737fe640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c034dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f976c00b1a0 tx=0x7f976402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d46e35abcfe6f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 1 -- 172.21.15.70:0/2136361296 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764037070 con 0x7f976c0091d0 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: dump: 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.259 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f4fbb3c5640 1 Processor -- start 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f4fbb3c5640 1 -- start start 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f4fbb3c5640 10 monclient: init 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T03:49:59.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4fb414a960 0x7f4fb40aa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7efd6dc3c640 1 -- start start 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7efd6dc3c640 10 monclient: init 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c09d490 con 0x7f611c14aa70 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f611c14b100 con 0x7f611c09bbe0 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f611c14ae50 con 0x7f611c09e060 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122b99640 10 monclient(hunting): authenticate will time out at 757.875333s 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 1 --2- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): get_auth_request con 0x7f611c09e060 auth_method 0 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7f6122398640 1 -- 172.21.15.70:0/3775732971 >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 msgr2=0x7f611c09db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.212+0000 7f6121396640 1 --2- 172.21.15.70:0/3775732971 >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 0x7f611c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-a con 0x7f4fb414a960 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f4fb414b350 0x7f4fb40ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-c con 0x7f4fb414b350 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fb40aac30 addr v1:172.21.15.70:6789/0 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbabc4640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4fb40aac30 0x7f4fb41471f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33754/0 (socket says 172.21.15.70:33754) 2024-09-16T03:49:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbabc4640 1 -- 172.21.15.70:0/620657765 learned_addr learned my addr 172.21.15.70:0/620657765 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fba3c3640 1 --2- 172.21.15.70:0/620657765 >> v2:172.21.15.70:3300/0 conn(0x7f4fb414a960 0x7f4fb40aa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd680712d0 addr v1:172.21.15.163:6789/0 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efd68070060 0x7efd68072e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-a con 0x7efd68070060 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd6814a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-b con 0x7efd68073340 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd68072870 con 0x7efd680712d0 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efd680716b0 con 0x7efd68070060 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7efd68070470 con 0x7efd68073340 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd6dc3c640 10 monclient(hunting): authenticate will time out at 757.875801s 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7efd677fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efd680712d0 0x7efd68143780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.70:36110/0 (socket says 172.21.15.70:36110) 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd677fe640 1 -- 172.21.15.70:0/981090097 learned_addr learned my addr 172.21.15.70:0/981090097 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 1 --2- 172.21.15.70:0/981090097 >> v2:172.21.15.70:3300/0 conn(0x7efd68070060 0x7efd68072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd67fff640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd6814a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346111107 0 0) 0x7efd68072870 con 0x7efd680712d0 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient(hunting): my global_id is 4396 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50005730 con 0x7efd680712d0 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): get_auth_request con 0x7efd68070060 auth_method 0 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 1 --2- 172.21.15.70:0/3775732971 >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 0x7f611c09db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6120b95640 1 -- 172.21.15.70:0/3775732971 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590352074 0 0) 0x7f611c09d490 con 0x7f611c14aa70 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 1 -- 172.21.15.70:0/3775732971 >> v1:172.21.15.163:6789/0 conn(0x7f611c14aa70 legacy=0x7f611c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb40acad0 con 0x7f4fb40aac30 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f6122398640 1 -- 172.21.15.70:0/3775732971 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c09d700 con 0x7f611c09e060 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb40af0c0 con 0x7f4fb414a960 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb414ad40 con 0x7f4fb414b350 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f779ab11640 1 -- 172.21.15.70:0/3147468952 shutdown_connections 2024-09-16T03:49:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f4fbb3c5640 10 monclient(hunting): authenticate will time out at 757.876641s 2024-09-16T03:49:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 1 --2- 172.21.15.70:0/250991534 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 0x7fcdc40c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): get_auth_request con 0x7fcdc4030730 auth_method 0 2024-09-16T03:49:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.213+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 1 -- 172.21.15.70:0/250991534 >> v1:172.21.15.163:6789/0 conn(0x7fcdc4006c00 legacy=0x7fcdc402f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 1 -- 172.21.15.70:0/250991534 >> v1:172.21.15.70:6789/0 conn(0x7fcdc402c070 legacy=0x7fcdc40317d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fcdd1108640 1 -- 172.21.15.70:0/250991534 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc40312d0 con 0x7fcdc4030730 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fd233d54640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7fd233d54640 10 monclient: monmap: 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.271 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2357788644 0 0) 0x7efd50005730 con 0x7efd680712d0 2024-09-16T03:49:59.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 msgr2=0x7efd6814a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd6814a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 >> v2:172.21.15.70:3300/0 conn(0x7efd68070060 msgr2=0x7efd68072e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 --2- 172.21.15.70:0/981090097 >> v2:172.21.15.70:3300/0 conn(0x7efd68070060 0x7efd68072e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.163:6789/0 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd68070ba0 con 0x7efd680712d0 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f6121396640 10 monclient: get_auth_request con 0x7f611c09bbe0 auth_method 0 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7efd58001670 con 0x7efd680712d0 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: dump: 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.274 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] but i'm connected to v1:172.21.15.163:6789/0, reconnecting 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f6121396640 1 --2- 172.21.15.70:0/3775732971 >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 0x7f611c09db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 >> v1:172.21.15.163:6789/0 conn(0x7efd680712d0 legacy=0x7efd68143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd50005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): picked mon.c con 0x7efd68073340 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd50006100 0x7efd50006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): picked mon.b con 0x7efd50006100 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd50006f50 0x7efd500073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f6120b95640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): picked mon.a con 0x7efd50006f50 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 librados: starting msgr at 2024-09-16T03:49:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 librados: starting objecter 2024-09-16T03:49:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 Processor -- start 2024-09-16T03:49:59.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9773fff640 5 monclient: authenticate success, global_id 4359 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9773fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 1 -- 172.21.15.70:0/2136361296 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764032440 con 0x7f976c0091d0 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9773fff640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9773fff640 4 set_mon_vals no callback set 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 1 -- 172.21.15.70:0/2136361296 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764037070 con 0x7f976c0091d0 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.214+0000 7f9771ffb640 10 monclient: dump: 2024-09-16T03:49:59.278 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd680716b0 con 0x7efd50006f50 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd68070470 con 0x7efd50006100 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7efd5000bbf0 con 0x7efd68073340 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 10 monclient: shutdown 2024-09-16T03:49:59.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 -- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 msgr2=0x7f976c034dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c034dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f976c00b1a0 tx=0x7f976402a140 comp rx=0 tx=0).stop 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 -- 172.21.15.70:0/2136361296 shutdown_connections 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c034dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 --2- 172.21.15.70:0/2136361296 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c0cf3d0 0x7f976c034890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 --2- 172.21.15.70:0/2136361296 >> v2:172.21.15.163:3300/0 conn(0x7f976c006ee0 0x7f976c0cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): get_auth_request con 0x7f4fb414a960 auth_method 0 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd66ffd640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd50006100 0x7efd50006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd66ffd640 10 monclient(hunting): get_auth_request con 0x7efd50006100 auth_method 0 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f9773fff640 1 -- 172.21.15.70:0/2136361296 >> 172.21.15.70:0/2136361296 conn(0x7f976c005d60 msgr2=0x7f976c006170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd66ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 1 --2- 172.21.15.70:0/620657765 >> v2:172.21.15.163:3300/0 conn(0x7f4fb414b350 0x7f4fb40ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): get_auth_request con 0x7f4fb414b350 auth_method 0 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 1 -- 172.21.15.70:0/620657765 >> v2:172.21.15.163:3300/0 conn(0x7f4fb414b350 msgr2=0x7f4fb40ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 1 --2- 172.21.15.70:0/620657765 >> v2:172.21.15.163:3300/0 conn(0x7f4fb414b350 0x7f4fb40ae3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 1 -- 172.21.15.70:0/620657765 >> v1:172.21.15.70:6789/0 conn(0x7f4fb40aac30 legacy=0x7f4fb41471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 1 monclient: found mon.noname-a 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f4fba3c3640 1 -- 172.21.15.70:0/620657765 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb4073780 con 0x7f4fb414a960 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 -- start start 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 librados: setting wanted keys 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 1 librados: calling monclient init 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fd233d54640 10 monclient: init 2024-09-16T03:49:59.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f779ab11640 1 -- 172.21.15.70:0/3147468952 wait complete. 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6122398640 1 --2- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c151b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6114009d90 tx=0x7f611402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=315adf38447f25c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c0aaa40 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 0x7fd22c127d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c0adaa0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 0x7fd22c1282d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c12b9c0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd1108640 1 --2- 172.21.15.70:0/250991534 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 0x7fcdc40c74b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0097d0 tx=0x7fcdbc02a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f15553806db401 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 1 -- 172.21.15.70:0/250991534 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc037070 con 0x7fcdc4030730 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 10 monclient: dump: 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd0907640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd290b640 5 monclient: authenticate success, global_id 4322 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7fcdd290b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fcdd0907640 1 -- 172.21.15.70:0/250991534 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc02ae50 con 0x7fcdc4030730 2024-09-16T03:49:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fcdd0907640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 1 -- 172.21.15.70:0/3775732971 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114031600 con 0x7f611c09e060 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: dump: 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c0ab1b0 con 0x7fd22c12b9c0 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c0ab030 con 0x7fd22c0aaa40 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fd22c0684d0 con 0x7fd22c0adaa0 2024-09-16T03:49:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd233d54640 10 monclient(hunting): authenticate will time out at 757.878349s 2024-09-16T03:49:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 1 -- 172.21.15.70:0/3775732971 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61140317a0 con 0x7f611c09e060 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6122b99640 5 monclient: authenticate success, global_id 4319 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6122b99640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6122b99640 4 set_mon_vals no callback set 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 1 -- 172.21.15.70:0/3775732971 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114031b70 con 0x7f611c09e060 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd231af0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd231af0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34968/0 (socket says 172.21.15.70:34968) 2024-09-16T03:49:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd231af0640 1 -- 172.21.15.70:0/1861764448 learned_addr learned my addr 172.21.15.70:0/1861764448 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd231af0640 10 monclient(hunting): get_auth_request con 0x7fd22c0aaa40 auth_method 0 2024-09-16T03:49:59.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd2312ef640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 0x7fd22c127d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f4fba3c3640 1 --2- 172.21.15.70:0/620657765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414a960 0x7f4fb40aa6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b57f0 tx=0x7f4fb002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2279a27038b361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7f779ab11640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 10 monclient: monmap: 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fd231af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 librados: starting msgr at 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 librados: starting objecter 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 Processor -- start 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd231af0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 -- start start 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 librados: setting wanted keys 2024-09-16T03:49:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 1 librados: calling monclient init 2024-09-16T03:49:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f779ab11640 10 monclient: init 2024-09-16T03:49:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd231af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): get_auth_request con 0x7fd22c0adaa0 auth_method 0 2024-09-16T03:49:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f6120b95640 10 monclient: dump: 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.299 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.299 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.299 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.299 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd50005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): get_auth_request con 0x7efd68073340 auth_method 0 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.215+0000 7efd677fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 10 monclient(hunting): handle_auth_done global_id 4396 payload 293 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 -- 172.21.15.70:0/981090097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd50006100 msgr2=0x7efd50006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd50006100 0x7efd50006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 -- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd50006f50 msgr2=0x7efd500073d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd50006f50 0x7efd500073d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 monclient: found mon.c 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 1 -- 172.21.15.70:0/981090097 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd50005730 con 0x7efd68073340 2024-09-16T03:49:59.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7efd677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7efd6dc3c640 5 monclient: authenticate success, global_id 4396 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7efd6dc3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 10 monclient: shutdown 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fcdd290b640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7fcdd290b640 4 set_mon_vals no callback set 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 msgr2=0x7f611c151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 --2- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c151b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6114009d90 tx=0x7f611402a5d0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 10 monclient: shutdown 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 msgr2=0x7fcdc40c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 --2- 172.21.15.70:0/250991534 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 0x7fcdc40c74b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0097d0 tx=0x7fcdbc02a140 comp rx=0 tx=0).stop 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd231af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd231af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 1 -- 172.21.15.70:0/620657765 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0037070 con 0x7f4fb414a960 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: dump: 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 1 -- 172.21.15.70:0/620657765 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb002ad30 con 0x7f4fb414a960 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 1 -- 172.21.15.70:0/620657765 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0032660 con 0x7f4fb414a960 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fb93c1640 10 monclient: dump: 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.306 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fbb3c5640 5 monclient: authenticate success, global_id 4368 2024-09-16T03:49:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fbb3c5640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f4fbb3c5640 4 set_mon_vals no callback set 2024-09-16T03:49:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2322f1640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 0x7fd22c1282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 shutdown_connections 2024-09-16T03:49:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd231af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7efd677fe640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd50005bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd5800c580 tx=0x7efd50007a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4049a0a4427da3f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 shutdown_connections 2024-09-16T03:49:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 --2- 172.21.15.70:0/3775732971 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c151b30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2312ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 --2- 172.21.15.70:0/3775732971 >> v2:172.21.15.70:3300/0 conn(0x7f611c09bbe0 0x7f611c09db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 --2- 172.21.15.70:0/250991534 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4030730 0x7fcdc40c74b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7794136960 0x7f779413a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-b con 0x7f7794136960 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-c con 0x7f77941e08d0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f779ab11640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 0x7f7794137d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2322f1640 10 monclient(hunting): get_auth_request con 0x7fd22c12b9c0 auth_method 0 2024-09-16T03:49:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2322f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2322f1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f7793fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f7793fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45984/0 (socket says 172.21.15.70:45984) 2024-09-16T03:49:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fd2322f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 >> 172.21.15.70:0/250991534 conn(0x7fcdc4005a50 msgr2=0x7fcdc4005e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 >> 172.21.15.70:0/3775732971 conn(0x7f611c0b8a40 msgr2=0x7f611c0b8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 shutdown_connections 2024-09-16T03:49:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-09-16T03:49:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7fcdd290b640 1 -- 172.21.15.70:0/250991534 wait complete. 2024-09-16T03:49:59.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 -- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 msgr2=0x7fd22c127d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58007c80 con 0x7efd68073340 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: dump: 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.314 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd58034070 con 0x7efd68073340 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 1 -- 172.21.15.70:0/981090097 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58005d90 con 0x7efd68073340 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd667fc640 10 monclient: dump: 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.315 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd6dc3c640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7efd6dc3c640 4 set_mon_vals no callback set 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 0x7f7794137d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 shutdown_connections 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f6122b99640 1 -- 172.21.15.70:0/3775732971 wait complete. 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): picked mon.noname-a con 0x7f77941378c0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77940b59d0 con 0x7f77941378c0 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 1 -- 172.21.15.70:0/2712626245 learned_addr learned my addr 172.21.15.70:0/2712626245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 10 monclient: shutdown 2024-09-16T03:49:59.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414a960 msgr2=0x7f4fb40aa6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/620657765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414a960 0x7f4fb40aa6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40b57f0 tx=0x7f4fb002a0a0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77988ad640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7794136960 0x7f779413a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 0x7fd22c127d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 -- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 msgr2=0x7fd22c1282d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 0x7fd22c1282d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd231af0640 1 -- 172.21.15.70:0/1861764448 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22c12c4c0 con 0x7fd22c0aaa40 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77940b5850 con 0x7f7794136960 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f77940b5b50 con 0x7f77941e08d0 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f779ab11640 10 monclient(hunting): authenticate will time out at 757.880603s 2024-09-16T03:49:59.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.216+0000 7f9773fff640 1 -- 172.21.15.70:0/2136361296 shutdown_connections 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.217+0000 7f9773fff640 1 -- 172.21.15.70:0/2136361296 wait complete. 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 shutdown_connections 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/620657765 >> v2:172.21.15.163:3300/0 conn(0x7f4fb414b350 0x7f4fb40ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/620657765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414a960 0x7f4fb40aa6f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 >> 172.21.15.70:0/620657765 conn(0x7f4fb4068050 msgr2=0x7f4fb40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f9773fff640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f9773fff640 10 monclient: monmap: 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): get_auth_request con 0x7f77941e08d0 auth_method 0 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77988ad640 10 monclient(hunting): get_auth_request con 0x7f7794136960 auth_method 0 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77988ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77988ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd2312ef640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 0x7fd22c127d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:59.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fd2322f1640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 0x7fd22c1282d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77988ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): get_auth_request con 0x7f77941378c0 auth_method 0 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f7793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fcdd290b640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7fcdd290b640 10 monclient: monmap: 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.325 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 10 monclient: shutdown 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 -- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 msgr2=0x7efd50005bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/981090097 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd68073340 0x7efd50005bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd5800c580 tx=0x7efd50007a40 comp rx=0 tx=0).stop 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd67fff640 1 -- 172.21.15.70:0/981090097 reap_dead start 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f77990ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 -- 172.21.15.70:0/2712626245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7794136960 msgr2=0x7f779413a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7794136960 0x7f779413a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 -- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 msgr2=0x7f7794137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 shutdown_connections 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 librados: starting msgr at 2024-09-16T03:49:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 librados: starting objecter 2024-09-16T03:49:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/620657765 wait complete. 2024-09-16T03:49:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 Processor -- start 2024-09-16T03:49:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 -- 172.21.15.70:0/981090097 shutdown_connections 2024-09-16T03:49:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 -- 172.21.15.70:0/981090097 >> 172.21.15.70:0/981090097 conn(0x7efd680b5d40 msgr2=0x7efd680b6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 0x7f7794137d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779413c930 con 0x7f77941e08d0 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f77990ae640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 0x7f7794137d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:59.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 -- start start 2024-09-16T03:49:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 -- 172.21.15.70:0/981090097 shutdown_connections 2024-09-16T03:49:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7efd6dc3c640 1 -- 172.21.15.70:0/981090097 wait complete. 2024-09-16T03:49:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f7793fff640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7780005040 tx=0x7f7780001480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f77917fa640 10 client.?.objecter ms_handle_connect 0x7f77941e08d0 2024-09-16T03:49:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f77917fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fd231af0640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001eb0 tx=0x7fd21c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fd2227fc640 10 client.?.objecter ms_handle_connect 0x7fd22c0aaa40 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fd2227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 librados: setting wanted keys 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 1 librados: calling monclient init 2024-09-16T03:49:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7fcdd290b640 10 monclient: init 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f9773fff640 1 librados: starting msgr at 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.218+0000 7f9773fff640 1 librados: starting objecter 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f9773fff640 1 Processor -- start 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f9773fff640 1 -- start start 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f9773fff640 1 librados: setting wanted keys 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f9773fff640 1 librados: calling monclient init 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f9773fff640 10 monclient: init 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:59.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 10 monclient: monmap: 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.333 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 0x7fcdc40ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc4006c00 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 librados: starting msgr at 2024-09-16T03:49:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 librados: starting objecter 2024-09-16T03:49:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 Processor -- start 2024-09-16T03:49:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 0x7fcdc40ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778003f020 con 0x7f77941e08d0 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: dump: 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7780003bb0 con 0x7f77941e08d0 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778003b400 con 0x7f77941e08d0 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77917fa640 10 monclient: dump: 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f77927fc640 4 set_mon_vals no callback set 2024-09-16T03:49:59.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49310/0 (socket says 172.21.15.70:49310) 2024-09-16T03:49:59.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 0x7fcdc40ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46000/0 (socket says 172.21.15.70:46000) 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 1 -- 172.21.15.70:0/200631785 learned_addr learned my addr 172.21.15.70:0/200631785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc402c070 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdc4030730 0x7fcdc4029ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc4030730 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40d2d90 con 0x7fcdc402c070 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 10 monclient: monmap: 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 librados: starting msgr at 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 librados: starting objecter 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 Processor -- start 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 -- start start 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 librados: setting wanted keys 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 1 librados: calling monclient init 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.219+0000 7f6122b99640 10 monclient: init 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f611c09bbe0 0x7f611c070c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-a con 0x7f611c09bbe0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c1e4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-c con 0x7f611c09e060 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient(hunting): picked mon.noname-b con 0x7f611c14aa70 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f611c09e480 con 0x7f611c09bbe0 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f6122b99640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f611c0b9940 con 0x7f611c14aa70 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122b99640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122b99640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f611c09e600 con 0x7f611c09e060 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122b99640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122b99640 10 monclient(hunting): authenticate will time out at 757.883061s 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c1e4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c1e4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46024/0 (socket says 172.21.15.70:46024) 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 1 -- 172.21.15.70:0/4263141364 learned_addr learned my addr 172.21.15.70:0/4263141364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:34992/0 (socket says 172.21.15.70:34992) 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): get_auth_request con 0x7f611c09e060 auth_method 0 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6121396640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): get_auth_request con 0x7f611c14aa70 auth_method 0 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc402f7d0 con 0x7fcdc4030730 2024-09-16T03:49:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 5 monclient: authenticate success, global_id 4328 2024-09-16T03:49:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 10 monclient: _renew_subs 2024-09-16T03:49:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc40d2f10 con 0x7fcdc4006c00 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7efd6dc3c640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7efd6dc3c640 10 monclient: monmap: 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7efd6dc3c640 1 librados: starting msgr at 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7efd6dc3c640 1 librados: starting objecter 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7efd6dc3c640 1 Processor -- start 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7efd6dc3c640 1 -- start start 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7efd6dc3c640 1 librados: setting wanted keys 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7efd6dc3c640 1 librados: calling monclient init 2024-09-16T03:49:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7efd6dc3c640 10 monclient: init 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 -- start start 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 librados: setting wanted keys 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 1 librados: calling monclient init 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f4fbb3c5640 10 monclient: init 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f4fbb3c5640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f4fbb3c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f4fbb3c5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fb40aac30 0x7f4fb41d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-b con 0x7f4fb40aac30 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f4fbb3c5640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd290b640 10 monclient(hunting): authenticate will time out at 757.882811s 2024-09-16T03:49:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): get_auth_request con 0x7fcdc402c070 auth_method 0 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 -- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 msgr2=0x7f611c1e4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c1e4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 -- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f611c09bbe0 msgr2=0x7f611c070c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f611c09bbe0 0x7f611c070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f6122398640 1 -- 172.21.15.70:0/4263141364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c1e59d0 con 0x7f611c14aa70 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7794144310 con 0x7f77941e08d0 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 10 monclient: _renew_subs 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77941e03b0 con 0x7f77941e08d0 2024-09-16T03:49:59.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f779ab11640 1 librados: init done 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-c con 0x7f976c006ee0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:45998/0 (socket says 172.21.15.70:45998) 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 1 -- 172.21.15.70:0/1885207411 learned_addr learned my addr 172.21.15.70:0/1885207411 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-a con 0x7f976c0091d0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 0x7f976c16cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): picked mon.noname-b con 0x7f976c0cf3d0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f976c008430 con 0x7f976c0091d0 2024-09-16T03:49:59.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0082b0 con 0x7f976c0cf3d0 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f976c036720 con 0x7f976c006ee0 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9773fff640 10 monclient(hunting): authenticate will time out at 757.882775s 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 10 monclient(hunting): get_auth_request con 0x7f976c006ee0 auth_method 0 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f97737fe640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 0x7f976c16cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7f97737fe640 10 monclient(hunting): get_auth_request con 0x7f976c0cf3d0 auth_method 0 2024-09-16T03:49:59.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 10 monclient(hunting): get_auth_request con 0x7fcdc4006c00 auth_method 0 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1909640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.220+0000 7fcdd1108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 -- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 msgr2=0x7fcdc40ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 0x7fcdc40ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd210a640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdc4030730 0x7fcdc4029ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 -- 172.21.15.70:0/200631785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdc4030730 msgr2=0x7fcdc4029ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdc4030730 0x7fcdc4029ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 monclient: found mon.noname-a 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc40cb6b0 con 0x7fcdc402c070 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd68073340 0x7efd680ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-b con 0x7efd68073340 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdd1108640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009720 tx=0x7fcdbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdc27fc640 10 client.?.objecter ms_handle_connect 0x7fcdc402c070 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.221+0000 7fcdc27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c017070 con 0x7fd22c0aaa40 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd6813bf30 0x7efd6813c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-a con 0x7efd6813bf30 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): picked mon.noname-c con 0x7efd680aad80 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd68144010 con 0x7efd6813bf30 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd68143e90 con 0x7efd68073340 2024-09-16T03:49:59.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc042020 con 0x7fcdc402c070 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:59.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: dump: 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7efd68073720 con 0x7efd680aad80 2024-09-16T03:49:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd6dc3c640 10 monclient(hunting): authenticate will time out at 757.884216s 2024-09-16T03:49:59.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 5 monclient: authenticate success, global_id 4377 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 10 monclient: _renew_subs 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 10 monclient: dump: 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.363 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd2227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc4028e80 con 0x7fcdc402c070 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc038040 con 0x7fcdc402c070 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc037070 con 0x7fcdc402c070 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc27fc640 10 monclient: dump: 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.365 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46042/0 (socket says 172.21.15.70:46042) 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 -- 172.21.15.70:0/3467653096 learned_addr learned my addr 172.21.15.70:0/3467653096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): get_auth_request con 0x7efd680aad80 auth_method 0 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fba3c3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fb40aac30 0x7f4fb41d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fb9bc2640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fb9bc2640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46034/0 (socket says 172.21.15.70:46034) 2024-09-16T03:49:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd6813bf30 0x7efd6813c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fb9bc2640 1 -- 172.21.15.70:0/661255548 learned_addr learned my addr 172.21.15.70:0/661255548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 5 monclient: authenticate success, global_id 4402 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 10 monclient: _renew_subs 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 1 -- 172.21.15.70:0/1861764448 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd22c12c730 con 0x7fd22c0aaa40 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 10 monclient: _renew_subs 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 1 -- 172.21.15.70:0/1861764448 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd22c12ae40 con 0x7fd22c0aaa40 2024-09-16T03:49:59.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fd233d54640 1 librados: init done 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-c con 0x7f4fb414a960 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414b350 0x7f4fb41d3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): get_auth_request con 0x7efd6813bf30 auth_method 0 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 -- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 msgr2=0x7f976c030c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97727fc640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f6122398640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6114031000 tx=0x7f6114004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f61127fc640 10 client.?.objecter ms_handle_connect 0x7f611c14aa70 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f61127fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 -- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 msgr2=0x7f976c0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c0cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7f97737fe640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c031690 con 0x7f976c0cf3d0 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f9772ffd640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f97727fc640 10 monclient: get_auth_request con 0x7f976c0091d0 auth_method 0 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f97727fc640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f97737fe640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 0x7f976c16cdc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9764031000 tx=0x7f97640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f975b7fe640 10 client.?.objecter ms_handle_connect 0x7f976c0cf3d0 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f975b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 -- 172.21.15.70:0/3467653096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd68073340 msgr2=0x7efd680ae130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T03:49:59.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd68073340 0x7efd680ae130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 -- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd6813bf30 msgr2=0x7efd6813c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd6813bf30 0x7efd6813c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7efd67fff640 1 -- 172.21.15.70:0/3467653096 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd680af3e0 con 0x7efd680aad80 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7efd67fff640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd680b01b0 tx=0x7efd6000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7efd4ffff640 10 client.?.objecter ms_handle_connect 0x7efd680aad80 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7efd4ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114040070 con 0x7f611c14aa70 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: dump: 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.375 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6114003b90 con 0x7f611c14aa70 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114030960 con 0x7f611c14aa70 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f61127fc640 10 monclient: dump: 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.376 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): picked mon.noname-a con 0x7f4fb414b350 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb414aec0 con 0x7f4fb414b350 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbabc4640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414b350 0x7f4fb41d3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb4072ef0 con 0x7f4fb40aac30 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb40ae920 con 0x7f4fb414a960 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbb3c5640 10 monclient(hunting): authenticate will time out at 757.885467s 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): get_auth_request con 0x7f4fb414a960 auth_method 0 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbabc4640 10 monclient(hunting): get_auth_request con 0x7f4fb414b350 auth_method 0 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbabc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbabc4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fbabc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 5 monclient: authenticate success, global_id 4408 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 10 monclient: _renew_subs 2024-09-16T03:49:59.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611c0ae6e0 con 0x7f611c14aa70 2024-09-16T03:49:59.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 10 monclient: _renew_subs 2024-09-16T03:49:59.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f611c070160 con 0x7f611c14aa70 2024-09-16T03:49:59.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-16T03:49:59.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 -- 172.21.15.70:0/661255548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fb40aac30 msgr2=0x7f4fb41d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fb40aac30 0x7f4fb41d0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 -- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414b350 msgr2=0x7f4fb41d3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414b350 0x7f4fb41d3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f4fb9bc2640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb41d2db0 con 0x7f4fb414a960 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f779ab11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f779ab11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f779ab11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7794143040 con 0x7f77941e08d0 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7fd217fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7fd217fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7fd217fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 10 monclient: _renew_subs 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc40351a0 con 0x7fcdc402c070 2024-09-16T03:49:59.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.223+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c005080 con 0x7fd22c0aaa40 2024-09-16T03:49:59.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f6122b99640 1 librados: init done 2024-09-16T03:49:59.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdd290b640 1 librados: init done 2024-09-16T03:49:59.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.222+0000 7fcdc37fe640 4 set_mon_vals no callback set 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21c0052d0 con 0x7fd22c0aaa40 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fd2227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c003810 con 0x7fd22c0aaa40 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fd223fff640 4 set_mon_vals no callback set 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764042070 con 0x7f976c0cf3d0 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: dump: 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 5 monclient: authenticate success, global_id 4411 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 10 monclient: _renew_subs 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976c0cc540 con 0x7f976c0cf3d0 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f4fb9bc2640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800ebe0 tx=0x7f4fa800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f4fa6ffd640 10 client.?.objecter ms_handle_connect 0x7f4fb414a960 2024-09-16T03:49:59.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f4fa6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fcdaffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fcdaffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fcdaffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda8005080 con 0x7fcdc402c070 2024-09-16T03:49:59.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f6113fff640 4 set_mon_vals no callback set 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 10 monclient: _renew_subs 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976c0cb1f0 con 0x7f976c0cf3d0 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9764030650 con 0x7f976c0cf3d0 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f9773fff640 1 librados: init done 2024-09-16T03:49:59.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f6122b99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9764037070 con 0x7f976c0cf3d0 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f975b7fe640 10 monclient: dump: 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.390 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.391 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.224+0000 7f97717fa640 4 set_mon_vals no callback set 2024-09-16T03:49:59.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f6122b99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f6122b99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611c09db20 con 0x7f611c14aa70 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7fd2227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7fd2227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7fd2227fc640 10 monclient: dump: 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.392 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f9773fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f9773fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f9773fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.225+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976c0cee90 con 0x7f976c0cf3d0 2024-09-16T03:49:59.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f6114039040 con 0x7f611c14aa70 2024-09-16T03:49:59.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fcdbc045020 con 0x7fcdc402c070 2024-09-16T03:49:59.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7f61127fc640 1 --2- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 0x7f61000701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fd21c01a070 con 0x7fd22c0aaa40 2024-09-16T03:49:59.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7f6121b97640 1 --2- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 0x7f61000701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7f6121b97640 10 monclient: get_auth_request con 0x7f610006dd00 auth_method 0 2024-09-16T03:49:59.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7fcdc27fc640 1 --2- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 0x7fcdb0078920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.226+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f61140aba50 con 0x7f611c14aa70 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f61127fc640 10 client.4408.objecter ms_dispatch 0x7f611c000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f61127fc640 3 client.4408.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f61127fc640 3 client.4408.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f6121b97640 1 --2- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 0x7f61000701c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61180015c0 tx=0x7f61180078b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f61127fc640 20 client.4408.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f61127fc640 10 client.4408.objecter ms_handle_connect 0x7f610006dd00 2024-09-16T03:49:59.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdd1909640 1 --2- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 0x7fcdb0078920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdd1909640 10 monclient: get_auth_request con 0x7fcdb0076460 auth_method 0 2024-09-16T03:49:59.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fd2227fc640 1 --2- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 0x7fd210070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6000ae00 con 0x7efd680aad80 2024-09-16T03:49:59.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fd2312ef640 1 --2- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 0x7fd210070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 10 monclient: dump: 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.398 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 5 monclient: authenticate success, global_id 4340 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 10 monclient: _renew_subs 2024-09-16T03:49:59.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdd1909640 1 --2- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 0x7fcdb0078920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80011e0 tx=0x7fcdc8007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fcdbc0abd60 con 0x7fcdc402c070 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd681c9390 con 0x7efd680aad80 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 10 monclient: _renew_subs 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd680ae970 con 0x7efd680aad80 2024-09-16T03:49:59.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7efd6dc3c640 1 librados: init done 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 10 client.4377.objecter ms_dispatch 0x7fcdcc000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 3 client.4377.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 3 client.4377.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 20 client.4377.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7fcdc27fc640 10 client.4377.objecter ms_handle_connect 0x7fcdb0076460 2024-09-16T03:49:59.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2312ef640 10 monclient: get_auth_request con 0x7fd21006dcd0 auth_method 0 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f7780042020 con 0x7f77941e08d0 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.227+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8017070 con 0x7f4fb414a960 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 10 monclient: dump: 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.402 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 5 monclient: authenticate success, global_id 4343 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 10 monclient: _renew_subs 2024-09-16T03:49:59.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb41d3090 con 0x7f4fb414a960 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 10 monclient: _renew_subs 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb41d1720 con 0x7f4fb414a960 2024-09-16T03:49:59.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f4fbb3c5640 1 librados: init done 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7fd21c08a900 con 0x7fd22c0aaa40 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2227fc640 10 client.4402.objecter ms_dispatch 0x7fd22c000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2227fc640 3 client.4402.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2227fc640 3 client.4402.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2227fc640 20 client.4402.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 1 --2- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 0x7f7778070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7fd2312ef640 1 --2- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 0x7fd210070190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd22c141e90 tx=0x7fd22800a1a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f77800ab190 con 0x7f77941e08d0 2024-09-16T03:49:59.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7fd2227fc640 10 client.4402.objecter ms_handle_connect 0x7fd21006dcd0 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77988ad640 1 --2- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 0x7f7778070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 10 client.4328.objecter ms_dispatch 0x7f7794000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 3 client.4328.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77917fa640 3 client.4328.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.228+0000 7f77988ad640 10 monclient: get_auth_request con 0x7f777806dcd0 auth_method 0 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f77917fa640 20 client.4328.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7efd6dc3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f77988ad640 1 --2- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 0x7f7778070190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7788006fd0 tx=0x7f7788008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7efd6dc3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7efd6dc3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd68075e80 con 0x7efd680aad80 2024-09-16T03:49:59.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f4fbb3c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f77917fa640 10 client.4328.objecter ms_handle_connect 0x7f777806dcd0 2024-09-16T03:49:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f4fbb3c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f4fbb3c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb41d3260 con 0x7f4fb414a960 2024-09-16T03:49:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd21c021070 con 0x7fd22c0aaa40 2024-09-16T03:49:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7fd2227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.229+0000 7fd2227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.231+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f9764045070 con 0x7f976c0cf3d0 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.231+0000 7f975b7fe640 1 --2- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 0x7f9760070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.231+0000 7f9772ffd640 1 --2- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 0x7f9760070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.231+0000 7f9772ffd640 10 monclient: get_auth_request con 0x7f976006dc50 auth_method 0 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.232+0000 7f975b7fe640 10 client.4411.objecter ms_handle_connect 0x7f976006dc50 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.232+0000 7f9772ffd640 1 --2- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 0x7f9760070110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9774074660 tx=0x7f977406f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.232+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcdbc04c030 con 0x7fcdc402c070 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.232+0000 7fcdc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.232+0000 7fcdc27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f9764077be0 con 0x7f976c0cf3d0 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f975b7fe640 10 client.4411.objecter ms_dispatch 0x7f9774000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f975b7fe640 3 client.4411.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f975b7fe640 3 client.4411.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f975b7fe640 20 client.4411.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f611c1e59d0 con 0x7f611c14aa70 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f61127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f61127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa800ecd0 con 0x7f4fb414a960 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8021060 con 0x7f4fb414a960 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f4fa6ffd640 10 monclient: dump: 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.414 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60017070 con 0x7efd680aad80 2024-09-16T03:49:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60003b90 con 0x7efd680aad80 2024-09-16T03:49:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7780049030 con 0x7f77941e08d0 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f77917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7f77917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd4ffff640 10 monclient: dump: 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.416 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.234+0000 7efd65ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.235+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f4fa801a070 con 0x7f4fb414a960 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 1 --2- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 0x7f4f980700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa7fff640 4 set_mon_vals no callback set 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f4fa808b560 con 0x7f4fb414a960 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 10 client.4343.objecter ms_dispatch 0x7f4fb4000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 3 client.4343.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 3 client.4343.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7f4fa6ffd640 20 client.4343.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.236+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7efd6001b410 con 0x7efd680aad80 2024-09-16T03:49:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7f4fba3c3640 1 --2- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 0x7f4f980700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7f4fba3c3640 10 monclient: get_auth_request con 0x7f4f9806dc00 auth_method 0 2024-09-16T03:49:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 1 --2- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 0x7efd540700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7efd6008adb0 con 0x7efd680aad80 2024-09-16T03:49:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 10 client.4340.objecter ms_dispatch 0x7efd68000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 3 client.4340.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 3 client.4340.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd4ffff640 20 client.4340.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7f4fa6ffd640 10 client.4343.objecter ms_handle_connect 0x7f4f9806dc00 2024-09-16T03:49:59.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7f4fba3c3640 1 --2- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 0x7f4f980700c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0034000 tx=0x7f4fb002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd677fe640 1 --2- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 0x7efd540700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.237+0000 7efd677fe640 10 monclient: get_auth_request con 0x7efd5406dc00 auth_method 0 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.238+0000 7efd677fe640 1 --2- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 0x7efd540700c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd58033000 tx=0x7efd58009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.238+0000 7efd4ffff640 10 client.4340.objecter ms_handle_connect 0x7efd5406dc00 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.238+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f976407a240 con 0x7f976c0cf3d0 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.238+0000 7f975b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.238+0000 7f975b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.244+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4fa805a350 con 0x7f4fb414a960 2024-09-16T03:49:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.244+0000 7f4fa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.244+0000 7f4fa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.253+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd60020020 con 0x7efd680aad80 2024-09-16T03:49:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.253+0000 7efd4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.253+0000 7efd4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.573+0000 7fbbece53640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.573+0000 7fbbece53640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.573+0000 7fbbece53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.573+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbbcc005370 con 0x7fbbe8144d90 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.574+0000 7fbbd6ffd640 1 -- 172.21.15.70:0/78916369 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbbd8059b80 con 0x7fbbe8144d90 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.574+0000 7fbbd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:49:59.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.574+0000 7fbbd6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.578 INFO:teuthology.orchestra.run.smithi070.stdout:64424509445 2024-09-16T03:49:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.577+0000 7fbbece53640 10 librados: watch_flush enter 2024-09-16T03:49:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.577+0000 7fbbece53640 10 librados: watch_flush exit 2024-09-16T03:49:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 10 client.4313.objecter successfully canceled tick 2024-09-16T03:49:59.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 msgr2=0x7fbbc80701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 0x7fbbc80701c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc034000 tx=0x7fbbdc02e000 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 10 monclient: shutdown 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 msgr2=0x7fbbe8146380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b3e0 tx=0x7fbbd800b8a0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 shutdown_connections 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> v2:172.21.15.70:6804/198622594 conn(0x7fbbc806dd00 0x7fbbc80701c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbe814af80 0x7fbbe81468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbbe8144d90 0x7fbbe8146380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 --2- 172.21.15.70:0/78916369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbbe80b2f60 0x7fbbe8137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 >> 172.21.15.70:0/78916369 conn(0x7fbbe80679c0 msgr2=0x7fbbe80b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 shutdown_connections 2024-09-16T03:49:59.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.578+0000 7fbbece53640 1 -- 172.21.15.70:0/78916369 wait complete. 2024-09-16T03:49:59.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.579+0000 7fbbece53640 1 librados: shutdown 2024-09-16T03:49:59.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509445 for osd.4 2024-09-16T03:49:59.597 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.610+0000 7fd217fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.610+0000 7fd217fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.610+0000 7fd217fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.610+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd20c005640 con 0x7fd22c0aaa40 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.611+0000 7fd2227fc640 1 -- 172.21.15.70:0/1861764448 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd21c059010 con 0x7fd22c0aaa40 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.611+0000 7fd2227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.611+0000 7fd2227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.614 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:59.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 10 librados: watch_flush enter 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 10 librados: watch_flush exit 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 20 client.4402.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 10 client.4402.objecter successfully canceled tick 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 msgr2=0x7fd210070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 0x7fd210070190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd22c141e90 tx=0x7fd22800a1a0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 10 monclient: shutdown 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 msgr2=0x7fd22c12a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001eb0 tx=0x7fd21c00c890 comp rx=0 tx=0).stop 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 shutdown_connections 2024-09-16T03:49:59.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> v2:172.21.15.70:6804/198622594 conn(0x7fd21006dcd0 0x7fd210070190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd22c12b9c0 0x7fd22c1282d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fd22c0adaa0 0x7fd22c127d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 --2- 172.21.15.70:0/1861764448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd22c0aaa40 0x7fd22c12a900 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.614+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 >> 172.21.15.70:0/1861764448 conn(0x7fd22c068020 msgr2=0x7fd22c0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.615+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 shutdown_connections 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.615+0000 7fd217fff640 1 -- 172.21.15.70:0/1861764448 wait complete. 2024-09-16T03:49:59.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.615+0000 7fd217fff640 1 librados: shutdown 2024-09-16T03:49:59.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.624+0000 7f779ab11640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:59.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.624+0000 7f779ab11640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:59.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.624+0000 7f779ab11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.624+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f77940b86f0 con 0x7f77941e08d0 2024-09-16T03:49:59.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.625+0000 7f77917fa640 1 -- 172.21.15.70:0/2712626245 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7794144310 con 0x7f77941e08d0 2024-09-16T03:49:59.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.625+0000 7f77917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:49:59.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.625+0000 7f77917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.628 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:59.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.626+0000 7f4fbb3c5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:59.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.626+0000 7f4fbb3c5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:59.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.626+0000 7f4fbb3c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.626+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4fb40aec50 con 0x7f4fb414a960 2024-09-16T03:49:59.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.627+0000 7f4fa6ffd640 1 -- 172.21.15.70:0/661255548 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4fa8059c50 con 0x7f4fb414a960 2024-09-16T03:49:59.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.627+0000 7f4fa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-09-16T03:49:59.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.627+0000 7f4fa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.630 INFO:teuthology.orchestra.run.smithi070.stdout:64424509446 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 10 librados: watch_flush enter 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 10 librados: watch_flush exit 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 20 client.4328.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 10 client.4328.objecter successfully canceled tick 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 msgr2=0x7f7778070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 0x7f7778070190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7788006fd0 tx=0x7f7788008040 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 10 monclient: shutdown 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 msgr2=0x7f7794137380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.628+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7780005040 tx=0x7f7780001480 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 shutdown_connections 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dcd0 0x7f7778070190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77941378c0 0x7f7794137d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f77941e08d0 0x7f7794137380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 --2- 172.21.15.70:0/2712626245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7794136960 0x7f779413a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 >> 172.21.15.70:0/2712626245 conn(0x7f77940679c0 msgr2=0x7f7794067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 shutdown_connections 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 -- 172.21.15.70:0/2712626245 wait complete. 2024-09-16T03:49:59.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f779ab11640 1 librados: shutdown 2024-09-16T03:49:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f4fbb3c5640 10 librados: watch_flush enter 2024-09-16T03:49:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f4fbb3c5640 10 librados: watch_flush exit 2024-09-16T03:49:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f4fbb3c5640 20 client.4343.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.629+0000 7f4fbb3c5640 10 client.4343.objecter successfully canceled tick 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 msgr2=0x7f4f980700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 0x7f4f980700c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0034000 tx=0x7f4fb002e000 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 10 monclient: shutdown 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 msgr2=0x7f4fb41d1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800ebe0 tx=0x7f4fa800c520 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 shutdown_connections 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> v2:172.21.15.70:6804/198622594 conn(0x7f4f9806dc00 0x7f4f980700c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fb414b350 0x7f4fb41d3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f4fb414a960 0x7f4fb41d1f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 --2- 172.21.15.70:0/661255548 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fb40aac30 0x7f4fb41d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 >> 172.21.15.70:0/661255548 conn(0x7f4fb4068050 msgr2=0x7f4fb40b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 shutdown_connections 2024-09-16T03:49:59.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 -- 172.21.15.70:0/661255548 wait complete. 2024-09-16T03:49:59.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.630+0000 7f4fbb3c5640 1 librados: shutdown 2024-09-16T03:49:59.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.3 2024-09-16T03:49:59.638 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdaffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdaffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdaffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcda8005640 con 0x7fcdc402c070 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdc27fc640 1 -- 172.21.15.70:0/200631785 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcdbc07ac50 con 0x7fcdc402c070 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:49:59.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.634+0000 7fcdc27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.639 INFO:teuthology.orchestra.run.smithi070.stdout:55834574853 2024-09-16T03:49:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.636+0000 7f9773fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.636+0000 7f9773fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.636+0000 7f9773fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.636+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f976c038b90 con 0x7f976c0cf3d0 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7f975b7fe640 1 -- 172.21.15.70:0/1885207411 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9764079b40 con 0x7f976c0cf3d0 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7f975b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7f975b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7fcdaffff640 10 librados: watch_flush enter 2024-09-16T03:49:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7f6122b99640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7fcdaffff640 10 librados: watch_flush exit 2024-09-16T03:49:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7f6122b99640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7fcdaffff640 20 client.4377.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7f6122b99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:49:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.637+0000 7fcdaffff640 10 client.4377.objecter successfully canceled tick 2024-09-16T03:49:59.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 msgr2=0x7fcdb0078920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 0x7fcdb0078920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80011e0 tx=0x7fcdc8007d30 comp rx=0 tx=0).stop 2024-09-16T03:49:59.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 10 monclient: shutdown 2024-09-16T03:49:59.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 msgr2=0x7fcdc4029580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009720 tx=0x7fcdbc0046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f611c0ae490 con 0x7f611c14aa70 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 shutdown_connections 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> v2:172.21.15.70:6804/198622594 conn(0x7fcdb0076460 0x7fcdb0078920 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdc4030730 0x7fcdc4029ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc402c070 0x7fcdc4029580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 --2- 172.21.15.70:0/200631785 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fcdc4006c00 0x7fcdc40ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 >> 172.21.15.70:0/200631785 conn(0x7fcdc4005a50 msgr2=0x7fcdc4005110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 shutdown_connections 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 -- 172.21.15.70:0/200631785 wait complete. 2024-09-16T03:49:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7fcdaffff640 1 librados: shutdown 2024-09-16T03:49:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7f61127fc640 1 -- 172.21.15.70:0/4263141364 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f611407a2b0 con 0x7f611c14aa70 2024-09-16T03:49:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7f61127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-09-16T03:49:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.638+0000 7f61127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.647 INFO:teuthology.orchestra.run.smithi070.stdout:60129542148 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 10 librados: watch_flush enter 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 10 librados: watch_flush exit 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 20 client.4411.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 10 client.4411.objecter successfully canceled tick 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 msgr2=0x7f9760070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 0x7f9760070110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9774074660 tx=0x7f977406f040 comp rx=0 tx=0).stop 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 10 monclient: shutdown 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 msgr2=0x7f976c16cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.639+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 0x7f976c16cdc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9764031000 tx=0x7f97640046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 shutdown_connections 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> v2:172.21.15.70:6804/198622594 conn(0x7f976006dc50 0x7f9760070110 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f976c0cf3d0 0x7f976c16cdc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f976c0091d0 0x7f976c0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 --2- 172.21.15.70:0/1885207411 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f976c006ee0 0x7f976c030c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 >> 172.21.15.70:0/1885207411 conn(0x7f976c005d60 msgr2=0x7f976c035610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 shutdown_connections 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 -- 172.21.15.70:0/1885207411 wait complete. 2024-09-16T03:49:59.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.640+0000 7f9773fff640 1 librados: shutdown 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 10 librados: watch_flush enter 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 10 librados: watch_flush exit 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 20 client.4408.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 10 client.4408.objecter successfully canceled tick 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 msgr2=0x7f61000701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 0x7f61000701c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61180015c0 tx=0x7f61180078b0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 10 monclient: shutdown 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.642+0000 7f6122b99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 msgr2=0x7f611c1e5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6114031000 tx=0x7f6114004a40 comp rx=0 tx=0).stop 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 shutdown_connections 2024-09-16T03:49:59.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> v2:172.21.15.70:6804/198622594 conn(0x7f610006dd00 0x7f61000701c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f611c14aa70 0x7f611c1e5490 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f611c09e060 0x7f611c1e4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 --2- 172.21.15.70:0/4263141364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f611c09bbe0 0x7f611c070c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 >> 172.21.15.70:0/4263141364 conn(0x7f611c0b8a40 msgr2=0x7f611c0b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 shutdown_connections 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 -- 172.21.15.70:0/4263141364 wait complete. 2024-09-16T03:49:59.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.643+0000 7f6122b99640 1 librados: shutdown 2024-09-16T03:49:59.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.6 2024-09-16T03:49:59.653 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2024-09-16T03:49:59.654 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.2 2024-09-16T03:49:59.658 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.659 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.7 2024-09-16T03:49:59.659 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-09-16T03:49:59.660 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.656+0000 7efd6dc3c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.656+0000 7efd6dc3c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.656+0000 7efd6dc3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.656+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7efd680ae790 con 0x7efd680aad80 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.657+0000 7efd4ffff640 1 -- 172.21.15.70:0/3467653096 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efd680ae790 con 0x7efd680aad80 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.657+0000 7efd4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.657+0000 7efd4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:59.661 INFO:teuthology.orchestra.run.smithi070.stdout:60129542149 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 10 librados: watch_flush enter 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 10 librados: watch_flush exit 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 20 client.4340.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 10 client.4340.objecter successfully canceled tick 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 msgr2=0x7efd540700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 0x7efd540700c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd58033000 tx=0x7efd58009030 comp rx=0 tx=0).stop 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 10 monclient: shutdown 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 msgr2=0x7efd680ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd680b01b0 tx=0x7efd6000cf20 comp rx=0 tx=0).stop 2024-09-16T03:49:59.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 shutdown_connections 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> v2:172.21.15.70:6804/198622594 conn(0x7efd5406dc00 0x7efd540700c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efd680aad80 0x7efd680ab200 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd6813bf30 0x7efd6813c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 --2- 172.21.15.70:0/3467653096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd68073340 0x7efd680ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 >> 172.21.15.70:0/3467653096 conn(0x7efd680b5d40 msgr2=0x7efd680b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 shutdown_connections 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 -- 172.21.15.70:0/3467653096 wait complete. 2024-09-16T03:49:59.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.659+0000 7efd6dc3c640 1 librados: shutdown 2024-09-16T03:49:59.672 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2024-09-16T03:49:59.673 DEBUG:teuthology.parallel:result is None 2024-09-16T03:49:59.673 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:49:59.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:49:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 10 monclient: get_monmap_and_config 2024-09-16T03:49:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 10 monclient: monmap: 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:49:59.809627+0000 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:49:59.809627+0000 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:49:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:49:59.812 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:49:59.812 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:49:59.812 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 1 Processor -- start 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 1 -- start start 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.808+0000 7feb553fd640 10 monclient: init 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7feb500b5650 0x7feb500b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-c con 0x7feb500b5650 addr v2:172.21.15.163:3300/0 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb50140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-b con 0x7feb50140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-a con 0x7feb50146500 addr v2:172.21.15.70:3300/0 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7feb500b5a30 con 0x7feb50146500 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7feb500aae40 con 0x7feb500b5650 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7feb500b5cd0 con 0x7feb50140410 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb553fd640 10 monclient(hunting): authenticate will time out at 758.471490s 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb50140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49352/0 (socket says 172.21.15.70:49352) 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 1 -- 172.21.15.70:0/1187038422 learned_addr learned my addr 172.21.15.70:0/1187038422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): get_auth_request con 0x7feb50140410 auth_method 0 2024-09-16T03:49:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): get_auth_request con 0x7feb50146500 auth_method 0 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4effd640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.163:3300/0 conn(0x7feb500b5650 0x7feb500b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4effd640 10 monclient(hunting): get_auth_request con 0x7feb500b5650 auth_method 0 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 -- 172.21.15.70:0/1187038422 >> v2:172.21.15.163:3300/0 conn(0x7feb500b5650 msgr2=0x7feb500b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.163:3300/0 conn(0x7feb500b5650 0x7feb500b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 -- 172.21.15.70:0/1187038422 >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 msgr2=0x7feb50146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:59.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 monclient: found mon.noname-b 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4e7fc640 1 -- 172.21.15.70:0/1187038422 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb500aa9f0 con 0x7feb50140410 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.809+0000 7feb4f7fe640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/1187038422 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb50140830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb500b6130 tx=0x7feb4402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b744b6f5237f4ca5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 1 -- 172.21.15.70:0/1187038422 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44037070 con 0x7feb50140410 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: dump: 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.826 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb553fd640 5 monclient: authenticate success, global_id 4346 2024-09-16T03:49:59.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 1 -- 172.21.15.70:0/1187038422 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4402add0 con 0x7feb50140410 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb553fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb553fd640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 1 -- 172.21.15.70:0/1187038422 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44032660 con 0x7feb50140410 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb553fd640 4 set_mon_vals no callback set 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.810+0000 7feb4dffb640 10 monclient: dump: 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.829 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 10 monclient: shutdown 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 -- 172.21.15.70:0/1187038422 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 msgr2=0x7feb50140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 --2- 172.21.15.70:0/1187038422 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb50140830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb500b6130 tx=0x7feb4402a080 comp rx=0 tx=0).stop 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 -- 172.21.15.70:0/1187038422 shutdown_connections 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.70:3300/0 conn(0x7feb50146500 0x7feb50146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 --2- 172.21.15.70:0/1187038422 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb50140830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 --2- 172.21.15.70:0/1187038422 >> v2:172.21.15.163:3300/0 conn(0x7feb500b5650 0x7feb500b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 -- 172.21.15.70:0/1187038422 >> 172.21.15.70:0/1187038422 conn(0x7feb500679c0 msgr2=0x7feb50067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 -- 172.21.15.70:0/1187038422 shutdown_connections 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.811+0000 7feb553fd640 1 -- 172.21.15.70:0/1187038422 wait complete. 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 10 monclient: build_initial_monmap 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 10 monclient: monmap: 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:49:59.831 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 librados: starting msgr at 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 librados: starting objecter 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 Processor -- start 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 -- start start 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 librados: setting wanted keys 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 1 librados: calling monclient init 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.812+0000 7feb553fd640 10 monclient: init 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:59.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb500b5650 0x7feb50145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-b con 0x7feb500b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-c con 0x7feb50140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 0x7feb501dc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): picked mon.noname-a con 0x7feb50146500 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feb50143000 con 0x7feb50146500 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feb501470e0 con 0x7feb500b5650 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7feb50143180 con 0x7feb50140410 2024-09-16T03:49:59.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient(hunting): authenticate will time out at 758.475249s 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46062/0 (socket says 172.21.15.70:46062) 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 -- 172.21.15.70:0/3821580063 learned_addr learned my addr 172.21.15.70:0/3821580063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): get_auth_request con 0x7feb50140410 auth_method 0 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 0x7feb501dc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): get_auth_request con 0x7feb50146500 auth_method 0 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb500b5650 0x7feb50145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): get_auth_request con 0x7feb500b5650 auth_method 0 2024-09-16T03:49:59.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 -- 172.21.15.70:0/3821580063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb500b5650 msgr2=0x7feb50145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb500b5650 0x7feb50145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 -- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 msgr2=0x7feb501dc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 0x7feb501dc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 monclient: found mon.noname-c 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 -- 172.21.15.70:0/3821580063 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb500b7480 con 0x7feb50140410 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4f7fe640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 0x7feb501dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4e7fc640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb44000c00 tx=0x7feb440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 client.?.objecter ms_handle_connect 0x7feb50140410 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44041070 con 0x7feb50140410 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: dump: 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.838 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4403c460 con 0x7feb50140410 2024-09-16T03:49:59.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 5 monclient: authenticate success, global_id 4349 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient: _renew_subs 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb501ce870 con 0x7feb50140410 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44037040 con 0x7feb50140410 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb377fe640 10 monclient: dump: 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:49:59.840 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 10 monclient: _renew_subs 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb501ced90 con 0x7feb50140410 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 1 librados: init done 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.813+0000 7feb4d7fa640 4 set_mon_vals no callback set 2024-09-16T03:49:59.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.814+0000 7feb553fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.815+0000 7feb553fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.815+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb50140830 con 0x7feb50140410 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.815+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7feb44044070 con 0x7feb50140410 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.815+0000 7feb377fe640 1 --2- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 0x7feb3c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb4effd640 1 --2- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 0x7feb3c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb4effd640 10 monclient: get_auth_request con 0x7feb3c06dc00 auth_method 0 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7feb440ab3d0 con 0x7feb50140410 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 10 client.4349.objecter ms_dispatch 0x7feb50000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 3 client.4349.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:49:59.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 3 client.4349.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:49:59.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 20 client.4349.objecter dump_active .. 0 homeless 2024-09-16T03:49:59.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb377fe640 10 client.4349.objecter ms_handle_connect 0x7feb3c06dc00 2024-09-16T03:49:59.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.816+0000 7feb4effd640 1 --2- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 0x7feb3c0700c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb40006fd0 tx=0x7feb40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:59.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.820+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7feb44079b00 con 0x7feb50140410 2024-09-16T03:49:59.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.820+0000 7feb377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:59.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.820+0000 7feb377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.985+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feb501cdc20 con 0x7feb3c06dc00 2024-09-16T03:49:59.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.987+0000 7feb377fe640 1 -- 172.21.15.70:0/3821580063 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35836 (secure 0 0 0) 0x7feb40010990 con 0x7feb3c06dc00 2024-09-16T03:49:59.990 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:49:59.990 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.988+0000 7feb553fd640 10 librados: watch_flush enter 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 10 librados: watch_flush exit 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 20 client.4349.objecter shutdown clearing up homeless session... 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 10 client.4349.objecter successfully canceled tick 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 msgr2=0x7feb3c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 0x7feb3c0700c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb40006fd0 tx=0x7feb40008040 comp rx=0 tx=0).stop 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 10 monclient: shutdown 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 msgr2=0x7feb501dc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb44000c00 tx=0x7feb440046c0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 shutdown_connections 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> v2:172.21.15.70:6804/198622594 conn(0x7feb3c06dc00 0x7feb3c0700c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb50146500 0x7feb501dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7feb50140410 0x7feb501dc460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 --2- 172.21.15.70:0/3821580063 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feb500b5650 0x7feb50145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 >> 172.21.15.70:0/3821580063 conn(0x7feb500679c0 msgr2=0x7feb50142330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 shutdown_connections 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 -- 172.21.15.70:0/3821580063 wait complete. 2024-09-16T03:49:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:49:59.989+0000 7feb553fd640 1 librados: shutdown 2024-09-16T03:50:00.002 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:49:58.633812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":231180,"kb_used_data":2468,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743540,"statfs":{"total":773094113280,"available":772857384960,"internally_reserved":0,"allocated":2527232,"data_stored":1539074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.793353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.907667+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T03:49:55.907667+0000","last_peered":"2024-09-16T03:49:55.907667+0000","last_clean":"2024-09-16T03:49:55.907667+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T03:49:55.907667+0000","last_undegraded":"2024-09-16T03:49:55.907667+0000","last_fullsized":"2024-09-16T03:49:55.907667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.861805+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T03:49:55.861805+0000","last_peered":"2024-09-16T03:49:55.861805+0000","last_clean":"2024-09-16T03:49:55.861805+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T03:49:55.861805+0000","last_undegraded":"2024-09-16T03:49:55.861805+0000","last_fullsized":"2024-09-16T03:49:55.861805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:00.954047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856793+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T03:49:55.856793+0000","last_peered":"2024-09-16T03:49:55.856793+0000","last_clean":"2024-09-16T03:49:55.856793+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T03:49:55.856793+0000","last_undegraded":"2024-09-16T03:49:55.856793+0000","last_fullsized":"2024-09-16T03:49:55.856793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.861804+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T03:49:55.861804+0000","last_peered":"2024-09-16T03:49:55.861804+0000","last_clean":"2024-09-16T03:49:55.861804+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T03:49:55.861804+0000","last_undegraded":"2024-09-16T03:49:55.861804+0000","last_fullsized":"2024-09-16T03:49:55.861804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:20.389723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.862729+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T03:49:55.862729+0000","last_peered":"2024-09-16T03:49:55.862729+0000","last_clean":"2024-09-16T03:49:55.862729+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T03:49:55.862729+0000","last_undegraded":"2024-09-16T03:49:55.862729+0000","last_fullsized":"2024-09-16T03:49:55.862729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:56.127656+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T03:49:56.127656+0000","last_peered":"2024-09-16T03:49:56.127656+0000","last_clean":"2024-09-16T03:49:56.127656+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T03:49:56.127656+0000","last_undegraded":"2024-09-16T03:49:56.127656+0000","last_fullsized":"2024-09-16T03:49:56.127656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856772+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T03:49:55.856772+0000","last_peered":"2024-09-16T03:49:55.856772+0000","last_clean":"2024-09-16T03:49:55.856772+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T03:49:55.856772+0000","last_undegraded":"2024-09-16T03:49:55.856772+0000","last_fullsized":"2024-09-16T03:49:55.856772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.859755+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T03:49:55.859755+0000","last_peered":"2024-09-16T03:49:55.859755+0000","last_clean":"2024-09-16T03:49:55.859755+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T03:49:55.859755+0000","last_undegraded":"2024-09-16T03:49:55.859755+0000","last_fullsized":"2024-09-16T03:49:55.859755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856269+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T03:49:55.856269+0000","last_peered":"2024-09-16T03:49:55.856269+0000","last_clean":"2024-09-16T03:49:55.856269+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T03:49:55.856269+0000","last_undegraded":"2024-09-16T03:49:55.856269+0000","last_fullsized":"2024-09-16T03:49:55.856269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":590243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":61334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":163840,"data_stored":51975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":61334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":590243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:50:00.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.098+0000 7f29e90af640 10 monclient: get_monmap_and_config 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.098+0000 7f29e90af640 10 monclient: build_initial_monmap 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.098+0000 7f29e90af640 10 monclient: monmap: 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:50:00.100300+0000 2024-09-16T03:50:00.101 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:50:00.100300+0000 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:50:00.102 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.099+0000 7f29e90af640 1 Processor -- start 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.099+0000 7f29e90af640 1 -- start start 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.099+0000 7f29e90af640 10 monclient: init 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 0x7f29e413cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40b5ba0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:00.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f29e40aab40 0x7f29e4146550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-c con 0x7f29e40aab40 addr v2:172.21.15.163:3300/0 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29e4146a90 addr v1:172.21.15.70:6789/0 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29e40abcf0 con 0x7f29e4146a90 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f29e40b3620 con 0x7f29e40aab40 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f29e40b5f80 con 0x7f29e40b5ba0 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e90af640 10 monclient(hunting): authenticate will time out at 758.762150s 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e359e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f29e4146a90 0x7f29e40b38a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33762/0 (socket says 172.21.15.70:33762) 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 0x7f29e413cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e359e640 1 -- 172.21.15.70:0/3160537399 learned_addr learned my addr 172.21.15.70:0/3160537399 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): get_auth_request con 0x7f29e40b5ba0 auth_method 0 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e259c640 1 --2- 172.21.15.70:0/3160537399 >> v2:172.21.15.163:3300/0 conn(0x7f29e40aab40 0x7f29e4146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e259c640 10 monclient(hunting): get_auth_request con 0x7f29e40aab40 auth_method 0 2024-09-16T03:50:00.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e1d9b640 1 -- 172.21.15.70:0/3160537399 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280783615 0 0) 0x7f29e40abcf0 con 0x7f29e4146a90 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 -- 172.21.15.70:0/3160537399 >> v2:172.21.15.163:3300/0 conn(0x7f29e40aab40 msgr2=0x7f29e4146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 --2- 172.21.15.70:0/3160537399 >> v2:172.21.15.163:3300/0 conn(0x7f29e40aab40 0x7f29e4146550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 -- 172.21.15.70:0/3160537399 >> v1:172.21.15.70:6789/0 conn(0x7f29e4146a90 legacy=0x7f29e40b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 monclient: found mon.noname-b 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e2d9d640 1 -- 172.21.15.70:0/3160537399 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e40abf60 con 0x7f29e40b5ba0 2024-09-16T03:50:00.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.100+0000 7f29e1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e2d9d640 1 --2- 172.21.15.70:0/3160537399 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 0x7f29e413cc40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f29d40097a0 tx=0x7f29d402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372a3b958eb37599 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 1 -- 172.21.15.70:0/3160537399 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4037070 con 0x7f29e40b5ba0 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: dump: 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.114 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: _finish_auth 0 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 1 -- 172.21.15.70:0/3160537399 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d402ae50 con 0x7f29e40b5ba0 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 5 monclient: authenticate success, global_id 4352 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 4 set_mon_vals no callback set 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 1 -- 172.21.15.70:0/3160537399 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4032660 con 0x7f29e40b5ba0 2024-09-16T03:50:00.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e1d9b640 10 monclient: dump: 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 10 monclient: shutdown 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 1 -- 172.21.15.70:0/3160537399 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 msgr2=0x7f29e413cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 1 --2- 172.21.15.70:0/3160537399 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 0x7f29e413cc40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f29d40097a0 tx=0x7f29d402a100 comp rx=0 tx=0).stop 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 1 -- 172.21.15.70:0/3160537399 shutdown_connections 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.101+0000 7f29e90af640 1 --2- 172.21.15.70:0/3160537399 >> v2:172.21.15.163:3300/0 conn(0x7f29e40aab40 0x7f29e4146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 1 --2- 172.21.15.70:0/3160537399 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e40b5ba0 0x7f29e413cc40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 1 -- 172.21.15.70:0/3160537399 >> 172.21.15.70:0/3160537399 conn(0x7f29e406cb40 msgr2=0x7f29e406cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 1 -- 172.21.15.70:0/3160537399 shutdown_connections 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 1 -- 172.21.15.70:0/3160537399 wait complete. 2024-09-16T03:50:00.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 10 monclient: build_initial_monmap 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.102+0000 7f29e90af640 10 monclient: monmap: 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:50:00.118 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 librados: starting msgr at 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 librados: starting objecter 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 Processor -- start 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 -- start start 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 librados: setting wanted keys 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 librados: calling monclient init 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient: init 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-a con 0x7f29e40aab40 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 0x7f29e40b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40b5ba0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:50:00.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 0x7f29e40b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): picked mon.noname-c con 0x7f29e4146a90 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:00.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f29e413d180 con 0x7f29e40aab40 2024-09-16T03:50:00.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29e40b5490 con 0x7f29e40b5ba0 2024-09-16T03:50:00.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f29e413d300 con 0x7f29e4146a90 2024-09-16T03:50:00.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:00.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e90af640 10 monclient(hunting): authenticate will time out at 758.765936s 2024-09-16T03:50:00.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.103+0000 7f29e2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49372/0 (socket says 172.21.15.70:49372) 2024-09-16T03:50:00.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 -- 172.21.15.70:0/792826801 learned_addr learned my addr 172.21.15.70:0/792826801 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:50:00.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): get_auth_request con 0x7f29e40aab40 auth_method 0 2024-09-16T03:50:00.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 0x7f29e40b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): get_auth_request con 0x7f29e4146a90 auth_method 0 2024-09-16T03:50:00.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-16T03:50:00.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e259c640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 0x7f29e40b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 -- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 msgr2=0x7f29e40b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 0x7f29e40b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 -- 172.21.15.70:0/792826801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 msgr2=0x7f29e40b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 0x7f29e40b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 monclient: found mon.noname-a 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 -- 172.21.15.70:0/792826801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e40b1ba0 con 0x7f29e40aab40 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e259c640 10 monclient: get_auth_request con 0x7f29e40b5ba0 auth_method 0 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e359e640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 0x7f29e40b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e259c640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 0x7f29e40b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e2d9d640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29d4035040 tx=0x7f29d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 client.?.objecter ms_handle_connect 0x7f29e40aab40 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4041070 con 0x7f29e40aab40 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: dump: 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.129 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d403c670 con 0x7f29e40aab40 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 5 monclient: authenticate success, global_id 4395 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4037040 con 0x7f29e40aab40 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 10 monclient: _renew_subs 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e40b2700 con 0x7f29e40aab40 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29cb7fe640 10 monclient: dump: 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e159a640 4 set_mon_vals no callback set 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:00.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 10 monclient: _renew_subs 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e40af300 con 0x7f29e40aab40 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.104+0000 7f29e90af640 1 librados: init done 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.105+0000 7f29e90af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.105+0000 7f29e90af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.105+0000 7f29e90af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.105+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29e4146fb0 con 0x7f29e40aab40 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f29d403a070 con 0x7f29e40aab40 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 1 --2- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 0x7f29d00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29e259c640 1 --2- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 0x7f29d00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f29d40ab100 con 0x7f29e40aab40 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 10 client.4395.objecter ms_dispatch 0x7f29e4000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 3 client.4395.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:50:00.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 3 client.4395.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29e259c640 10 monclient: get_auth_request con 0x7f29d006dc00 auth_method 0 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.106+0000 7f29cb7fe640 20 client.4395.objecter dump_active .. 0 homeless 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.107+0000 7f29cb7fe640 10 client.4395.objecter ms_handle_connect 0x7f29d006dc00 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.107+0000 7f29e259c640 1 --2- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 0x7f29d00700c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29e40b2b50 tx=0x7f29d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.110+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f29e40abf60 con 0x7f29e40aab40 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.110+0000 7f29cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.110+0000 7f29cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:00.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.274+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f29e40b28d0 con 0x7f29d006dc00 2024-09-16T03:50:00.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.276+0000 7f29cb7fe640 1 -- 172.21.15.70:0/792826801 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35836 (secure 0 0 0) 0x7f29e40b28d0 con 0x7f29d006dc00 2024-09-16T03:50:00.279 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:50:00.279 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.277+0000 7f29e90af640 10 librados: watch_flush enter 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.277+0000 7f29e90af640 10 librados: watch_flush exit 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.277+0000 7f29e90af640 20 client.4395.objecter shutdown clearing up homeless session... 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.277+0000 7f29e90af640 10 client.4395.objecter successfully canceled tick 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 msgr2=0x7f29d00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 0x7f29d00700c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29e40b2b50 tx=0x7f29d8008040 comp rx=0 tx=0).stop 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 10 monclient: shutdown 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:00.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 msgr2=0x7f29e41d8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f29d4035040 tx=0x7f29d40046c0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 shutdown_connections 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> v2:172.21.15.70:6804/198622594 conn(0x7f29d006dc00 0x7f29d00700c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f29e4146a90 0x7f29e40b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29e40b5ba0 0x7f29e40b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 --2- 172.21.15.70:0/792826801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29e40aab40 0x7f29e41d8320 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 >> 172.21.15.70:0/792826801 conn(0x7f29e406cb40 msgr2=0x7f29e40b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 shutdown_connections 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 -- 172.21.15.70:0/792826801 wait complete. 2024-09-16T03:50:00.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.278+0000 7f29e90af640 1 librados: shutdown 2024-09-16T03:50:00.291 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:49:58.633812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":231180,"kb_used_data":2468,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754743540,"statfs":{"total":773094113280,"available":772857384960,"internally_reserved":0,"allocated":2527232,"data_stored":1539074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.793353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.907667+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T03:49:55.907667+0000","last_peered":"2024-09-16T03:49:55.907667+0000","last_clean":"2024-09-16T03:49:55.907667+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T03:49:55.907667+0000","last_undegraded":"2024-09-16T03:49:55.907667+0000","last_fullsized":"2024-09-16T03:49:55.907667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.861805+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T03:49:55.861805+0000","last_peered":"2024-09-16T03:49:55.861805+0000","last_clean":"2024-09-16T03:49:55.861805+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T03:49:55.861805+0000","last_undegraded":"2024-09-16T03:49:55.861805+0000","last_fullsized":"2024-09-16T03:49:55.861805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:00.954047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856793+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T03:49:55.856793+0000","last_peered":"2024-09-16T03:49:55.856793+0000","last_clean":"2024-09-16T03:49:55.856793+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T03:49:55.856793+0000","last_undegraded":"2024-09-16T03:49:55.856793+0000","last_fullsized":"2024-09-16T03:49:55.856793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.861804+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T03:49:55.861804+0000","last_peered":"2024-09-16T03:49:55.861804+0000","last_clean":"2024-09-16T03:49:55.861804+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T03:49:55.861804+0000","last_undegraded":"2024-09-16T03:49:55.861804+0000","last_fullsized":"2024-09-16T03:49:55.861804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:20.389723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.862729+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T03:49:55.862729+0000","last_peered":"2024-09-16T03:49:55.862729+0000","last_clean":"2024-09-16T03:49:55.862729+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T03:49:55.862729+0000","last_undegraded":"2024-09-16T03:49:55.862729+0000","last_fullsized":"2024-09-16T03:49:55.862729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:56.127656+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T03:49:56.127656+0000","last_peered":"2024-09-16T03:49:56.127656+0000","last_clean":"2024-09-16T03:49:56.127656+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T03:49:56.127656+0000","last_undegraded":"2024-09-16T03:49:56.127656+0000","last_fullsized":"2024-09-16T03:49:56.127656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856772+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T03:49:55.856772+0000","last_peered":"2024-09-16T03:49:55.856772+0000","last_clean":"2024-09-16T03:49:55.856772+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T03:49:55.856772+0000","last_undegraded":"2024-09-16T03:49:55.856772+0000","last_fullsized":"2024-09-16T03:49:55.856772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.859755+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T03:49:55.859755+0000","last_peered":"2024-09-16T03:49:55.859755+0000","last_clean":"2024-09-16T03:49:55.859755+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T03:49:55.859755+0000","last_undegraded":"2024-09-16T03:49:55.859755+0000","last_fullsized":"2024-09-16T03:49:55.859755+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T03:49:55.856269+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T03:49:55.856269+0000","last_peered":"2024-09-16T03:49:55.856269+0000","last_clean":"2024-09-16T03:49:55.856269+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T03:49:55.856269+0000","last_undegraded":"2024-09-16T03:49:55.856269+0000","last_fullsized":"2024-09-16T03:49:55.856269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":590243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":61334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":61315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":163840,"data_stored":51975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":188416,"data_stored":61334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":590243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:50:00.292 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:50:00.292 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T03:50:00.292 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T03:50:00.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.386+0000 7f155a6e4640 10 monclient: get_monmap_and_config 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.386+0000 7f155a6e4640 10 monclient: build_initial_monmap 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.386+0000 7f155a6e4640 10 monclient: monmap: 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:50:00.387811+0000 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:50:00.387811+0000 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.389 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T03:50:00.390 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T03:50:00.390 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T03:50:00.390 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T03:50:00.390 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T03:50:00.390 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 Processor -- start 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 -- start start 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient: init 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15540b5650 addr v1:172.21.15.163:6789/0 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f15540b3120 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1554146a40 addr v1:172.21.15.70:6789/0 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15540aae40 con 0x7f1554146a40 2024-09-16T03:50:00.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1554140b20 con 0x7f15540b5650 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f15540b5a30 con 0x7f15540b3120 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f155a6e4640 10 monclient(hunting): authenticate will time out at 759.049785s 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f1558c81640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1554146a40 0x7f155414d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33772/0 (socket says 172.21.15.70:33772) 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f1558c81640 1 -- 172.21.15.70:0/2742731743 learned_addr learned my addr 172.21.15.70:0/2742731743 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46092/0 (socket says 172.21.15.70:46092) 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 10 monclient(hunting): get_auth_request con 0x7f15540b3120 auth_method 0 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.387+0000 7f15537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 1 -- 172.21.15.70:0/2742731743 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551570228 0 0) 0x7f15540aae40 con 0x7f1554146a40 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 10 monclient(hunting): my global_id is 4398 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f1552ffd640 1 -- 172.21.15.70:0/2742731743 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f153c003520 con 0x7f1554146a40 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 1 -- 172.21.15.70:0/2742731743 >> v1:172.21.15.163:6789/0 conn(0x7f15540b5650 legacy=0x7f1554140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 1 -- 172.21.15.70:0/2742731743 >> v1:172.21.15.70:6789/0 conn(0x7f1554146a40 legacy=0x7f155414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient: _finish_hunting 0 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 1 monclient: found mon.noname-b 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 1 -- 172.21.15.70:0/2742731743 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15540b5cd0 con 0x7f15540b3120 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.388+0000 7f15537fe640 1 --2- 172.21.15.70:0/2742731743 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15480097d0 tx=0x7f154802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=803f44a6f26d5ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 1 -- 172.21.15.70:0/2742731743 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1548037070 con 0x7f15540b3120 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: dump: 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.401 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 1 -- 172.21.15.70:0/2742731743 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1548032440 con 0x7f15540b3120 2024-09-16T03:50:00.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 5 monclient: authenticate success, global_id 4358 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 4 set_mon_vals no callback set 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 1 -- 172.21.15.70:0/2742731743 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1548032810 con 0x7f15540b3120 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f1552ffd640 10 monclient: dump: 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.403 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 10 monclient: shutdown 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 -- 172.21.15.70:0/2742731743 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 msgr2=0x7f1554146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2742731743 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15480097d0 tx=0x7f154802a180 comp rx=0 tx=0).stop 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 -- 172.21.15.70:0/2742731743 shutdown_connections 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2742731743 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b3120 0x7f1554146500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 -- 172.21.15.70:0/2742731743 >> 172.21.15.70:0/2742731743 conn(0x7f15540679c0 msgr2=0x7f1554067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 -- 172.21.15.70:0/2742731743 shutdown_connections 2024-09-16T03:50:00.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.389+0000 7f155a6e4640 1 -- 172.21.15.70:0/2742731743 wait complete. 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 10 monclient: build_initial_monmap 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 10 monclient: monmap: 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 librados: starting msgr at 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 librados: starting objecter 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 Processor -- start 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 -- start start 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 librados: setting wanted keys 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 1 librados: calling monclient init 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.390+0000 7f155a6e4640 10 monclient: init 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-a con 0x7f15540b3120 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-c con 0x7f15540b5650 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 0x7f155413d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f1554146a40 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1554147030 con 0x7f15540b3120 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1554146eb0 con 0x7f1554146a40 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f15541471b0 con 0x7f15540b5650 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f155a6e4640 10 monclient(hunting): authenticate will time out at 759.053374s 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:46106/0 (socket says 172.21.15.70:46106) 2024-09-16T03:50:00.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 1 -- 172.21.15.70:0/2211865721 learned_addr learned my addr 172.21.15.70:0/2211865721 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): get_auth_request con 0x7f15540b5650 auth_method 0 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49388/0 (socket says 172.21.15.70:49388) 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): get_auth_request con 0x7f15540b3120 auth_method 0 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1558c81640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 0x7f155413d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 -- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 msgr2=0x7f155413cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 -- 172.21.15.70:0/2211865721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 msgr2=0x7f155413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 0x7f155413d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient: _finish_hunting 0 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 monclient: found mon.noname-a 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:00.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 -- 172.21.15.70:0/2211865721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15540ae850 con 0x7f15540b3120 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1558c81640 10 monclient: get_auth_request con 0x7f1554146a40 auth_method 0 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1558c81640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 0x7f155413d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f15537fe640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1553fff640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f154400c390 tx=0x7f154400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1550ff9640 10 client.?.objecter ms_handle_connect 0x7f15540b3120 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1550ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.391+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544017070 con 0x7f15540b3120 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: dump: 2024-09-16T03:50:00.410 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: _finish_auth 0 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 5 monclient: authenticate success, global_id 4401 2024-09-16T03:50:00.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15440052d0 con 0x7f15540b3120 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544003830 con 0x7f15540b3120 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 monclient: _renew_subs 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155413d980 con 0x7f15540b3120 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f15527fc640 4 set_mon_vals no callback set 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f1550ff9640 10 monclient: dump: 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.412 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 monclient: _renew_subs 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15540ac7b0 con 0x7f15540b3120 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 1 librados: init done 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.392+0000 7f155a6e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.393+0000 7f155a6e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.393+0000 7f155a6e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.393+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15540b4710 con 0x7f15540b3120 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.393+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f154401a070 con 0x7f15540b3120 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.393+0000 7f1550ff9640 1 --2- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 0x7f15400700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f15537fe640 1 --2- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 0x7f15400700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4146+0+0 (secure 0 0 0) 0x7f154408a080 con 0x7f15540b3120 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 10 client.4401.objecter ms_dispatch 0x7f1554000b90 osd_map(21..21 src has 1..21) 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 3 client.4401.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 3 client.4401.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f15537fe640 10 monclient: get_auth_request con 0x7f154006dc00 auth_method 0 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 20 client.4401.objecter dump_active .. 0 homeless 2024-09-16T03:50:00.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f15537fe640 1 --2- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 0x7f15400700c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1548000c00 tx=0x7f154802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.394+0000 7f1550ff9640 10 client.4401.objecter ms_handle_connect 0x7f154006dc00 2024-09-16T03:50:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.398+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f15540b4710 con 0x7f15540b3120 2024-09-16T03:50:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.398+0000 7f1550ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:00.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.398+0000 7f1550ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:00.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f155a6e4640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T03:50:00.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f155a6e4640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:50:00.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f155a6e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T03:50:00.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1554146500 con 0x7f15540b3120 2024-09-16T03:50:00.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f1550ff9640 1 -- 172.21.15.70:0/2211865721 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f1544058840 con 0x7f15540b3120 2024-09-16T03:50:00.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f1550ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:50:00.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.598+0000 7f1550ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:00.602 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T03:50:00.602 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 10 librados: watch_flush enter 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 10 librados: watch_flush exit 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 20 client.4401.objecter shutdown clearing up homeless session... 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 10 client.4401.objecter successfully canceled tick 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 msgr2=0x7f15400700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 0x7f15400700c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1548000c00 tx=0x7f154802e000 comp rx=0 tx=0).stop 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 10 monclient: shutdown 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 msgr2=0x7f15540ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f154400c390 tx=0x7f154400c850 comp rx=0 tx=0).stop 2024-09-16T03:50:00.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 shutdown_connections 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> v2:172.21.15.70:6804/198622594 conn(0x7f154006dc00 0x7f15400700c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1554146a40 0x7f155413d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f15540b5650 0x7f155413cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 --2- 172.21.15.70:0/2211865721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15540b3120 0x7f15540ad4e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 >> 172.21.15.70:0/2211865721 conn(0x7f15540679c0 msgr2=0x7f1554145aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 shutdown_connections 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.600+0000 7f155a6e4640 1 -- 172.21.15.70:0/2211865721 wait complete. 2024-09-16T03:50:00.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T03:50:00.601+0000 7f155a6e4640 1 librados: shutdown 2024-09-16T03:50:00.613 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T03:50:00.614 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T03:50:00.628 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T03:50:00.628 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T03:50:00.629 DEBUG:teuthology.orchestra.run.smithi070:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:50:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:50:00.644 INFO:teuthology.orchestra.run.smithi070.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T03:50:00.644 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:50:00.699 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T03:50:00.700 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T03:50:00.754 INFO:tasks.workunit:timeout=6h 2024-09-16T03:50:00.754 INFO:tasks.workunit:cleanup=True 2024-09-16T03:50:00.755 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T03:50:00.810 INFO:tasks.workunit.client.0.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T03:50:56.194 INFO:tasks.workunit.client.0.smithi070.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T03:50:56.194 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.194 INFO:tasks.workunit.client.0.smithi070.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: git switch -c 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:Or undo this operation with: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: git switch - 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.195 INFO:tasks.workunit.client.0.smithi070.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T03:50:56.196 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-16T03:50:56.196 INFO:tasks.workunit.client.0.smithi070.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T03:50:56.198 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T03:50:56.276 INFO:tasks.workunit.client.0.smithi070.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T03:50:56.278 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:50:56.278 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T03:50:56.905 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T03:50:56.970 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T03:50:57.009 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:50:57.011 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:50:57.011 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T03:50:57.049 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:50:57.052 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-16T03:50:57.052 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T03:50:57.108 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T03:50:57.109 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T03:50:57.110 DEBUG:teuthology.orchestra.run.smithi070:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ parallel=1 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' '' = --serial ']' 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ crimson=0 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' '' = --crimson ']' 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ color= 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -t 1 ']' 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:50:57.172 INFO:tasks.workunit.client.0.smithi070.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:50:57.178 INFO:tasks.workunit.client.0.smithi070.stderr:+ declare -A pids 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_aio 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_aio' 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_aio 2024-09-16T03:50:57.179 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.180 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_aio 2024-09-16T03:50:57.180 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39545 2024-09-16T03:50:57.180 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_aio on pid 39545' 2024-09-16T03:50:57.181 INFO:tasks.workunit.client.0.smithi070.stdout:test api_aio on pid 39545 2024-09-16T03:50:57.183 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39545 2024-09-16T03:50:57.183 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.183 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.183 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T03:50:57.183 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_aio_pp 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_aio_pp' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_aio_pp 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.184 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.185 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.186 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.186 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_aio_pp 2024-09-16T03:50:57.186 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39552 2024-09-16T03:50:57.186 INFO:tasks.workunit.client.0.smithi070.stdout:test api_aio_pp on pid 39552 2024-09-16T03:50:57.189 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_aio_pp on pid 39552' 2024-09-16T03:50:57.189 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39552 2024-09-16T03:50:57.189 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_io 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_io' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_io 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.190 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.191 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.192 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.193 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.196 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_io 2024-09-16T03:50:57.197 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39561 2024-09-16T03:50:57.197 INFO:tasks.workunit.client.0.smithi070.stdout:test api_io on pid 39561 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_io on pid 39561' 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39561 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T03:50:57.201 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_io_pp 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_io_pp' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_io_pp 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.202 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.203 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.204 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.205 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.206 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_io_pp 2024-09-16T03:50:57.206 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39572 2024-09-16T03:50:57.206 INFO:tasks.workunit.client.0.smithi070.stdout:test api_io_pp on pid 39572 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_io_pp on pid 39572' 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39572 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_asio 2024-09-16T03:50:57.210 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_asio' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_asio 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.211 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.212 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.213 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.214 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.215 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_asio 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39586 2024-09-16T03:50:57.216 INFO:tasks.workunit.client.0.smithi070.stdout:test api_asio on pid 39586 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_asio on pid 39586' 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39586 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.222 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_list 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_list' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_list 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.223 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.224 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.225 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.226 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.227 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.228 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.229 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39545/exe 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_list 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39605 2024-09-16T03:50:57.230 INFO:tasks.workunit.client.0.smithi070.stdout:test api_list on pid 39605 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_list on pid 39605' 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39605 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.240 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_lock 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.241 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_lock' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.242 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.243 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_lock 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.244 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.245 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.246 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.247 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.248 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39552/exe 2024-09-16T03:50:57.249 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.250 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_lock 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.251 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39627 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.252 INFO:tasks.workunit.client.0.smithi070.stdout:test api_lock on pid 39627 2024-09-16T03:50:57.259 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.259 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_lock on pid 39627' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39627 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T03:50:57.260 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_lock_pp 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_lock_pp' 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.261 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39561/exe 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_lock_pp 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.262 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.263 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.264 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.265 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.266 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.267 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_lock_pp 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_io: /' 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.268 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39650 2024-09-16T03:50:57.269 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_lock_pp on pid 39650' 2024-09-16T03:50:57.269 INFO:tasks.workunit.client.0.smithi070.stdout:test api_lock_pp on pid 39650 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39650 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.280 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-16T03:50:57.281 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_misc 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.282 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39572/exe 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_misc' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.283 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.284 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.285 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.286 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39586/exe 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.287 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_misc 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.288 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T03:50:57.289 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T03:50:57.290 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-16T03:50:57.291 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_misc 2024-09-16T03:50:57.291 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39677 2024-09-16T03:50:57.291 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_misc on pid 39677' 2024-09-16T03:50:57.291 INFO:tasks.workunit.client.0.smithi070.stdout:test api_misc on pid 39677 2024-09-16T03:50:57.306 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39677 2024-09-16T03:50:57.306 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.306 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.306 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.307 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.308 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.309 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.310 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.311 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.312 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39605/exe 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.313 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.314 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.315 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.316 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.317 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_list: /' 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-16T03:50:57.318 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_misc_pp 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.319 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_misc_pp' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.320 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_misc_pp 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.321 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_misc_pp 2024-09-16T03:50:57.322 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39706 2024-09-16T03:50:57.322 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_misc_pp on pid 39706' 2024-09-16T03:50:57.322 INFO:tasks.workunit.client.0.smithi070.stdout:test api_misc_pp on pid 39706 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39706 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_tier_pp 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_tier_pp' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_tier_pp 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.326 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.327 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_tier_pp 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.328 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39713 2024-09-16T03:50:57.329 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_tier_pp on pid 39713' 2024-09-16T03:50:57.329 INFO:tasks.workunit.client.0.smithi070.stdout:test api_tier_pp on pid 39713 2024-09-16T03:50:57.338 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39713 2024-09-16T03:50:57.338 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.338 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T03:50:57.338 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.338 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_pool 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_pool' 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39627/exe 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.339 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.340 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.341 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.342 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.343 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.344 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_pool 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.345 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39677/exe 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.346 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_pool 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39771 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_pool on pid 39771' 2024-09-16T03:50:57.347 INFO:tasks.workunit.client.0.smithi070.stdout:test api_pool on pid 39771 2024-09-16T03:50:57.355 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39771 2024-09-16T03:50:57.355 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T03:50:57.355 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_snapshots 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_snapshots' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_snapshots 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.356 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.357 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.358 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-16T03:50:57.359 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.360 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.361 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_snapshots 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.362 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T03:50:57.363 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.259+0000 7fe3aec0f780 10 monclient: get_monmap_and_config 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.259+0000 7fe3aec0f780 10 monclient: build_initial_monmap 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.259+0000 7fe3aec0f780 10 monclient: monmap: 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 0 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 0.000000 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 0.000000 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 1 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:50:57.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.259+0000 7fe3aec0f780 1 Processor -- start 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 1 -- start start 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 10 monclient: init 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 10 monclient: _reopen_session rank -1 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7d7f0 0x55e033e17010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-a con 0x55e033d7d7f0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7f5e0 0x55e033d80800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-b con 0x55e033d7f5e0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.260+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-c con 0x55e033d847c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e033d7e6c0 con 0x55e033d7d7f0 2024-09-16T03:50:57.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e033d84ba0 con 0x55e033d7f5e0 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x55e033d7e9a0 con 0x55e033d847c0 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): _renew_subs 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3aec0f780 10 monclient(hunting): authenticate will time out at 815.922051s 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39819 2024-09-16T03:50:57.366 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_snapshots on pid 39819' 2024-09-16T03:50:57.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16test api_snapshots on pid 39819 2024-09-16T03:50:57.367 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39819 2024-09-16T03:50:57.367 INFO:tasks.workunit.client.0.smithi070.stdout:T03:50:57.261+0000 7fe3ac296640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7f5e0 0x55e033d80800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:43022/0 (socket says 172.21.15.70:43022) 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 -- 172.21.15.70:0/1930235706 learned_addr learned my addr 172.21.15.70:0/1930235706 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): get_auth_request con 0x55e033d847c0 auth_method 0 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): get_auth_request con 0x55e033d7f5e0 auth_method 0 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:57.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 -- 172.21.15.70:0/1930235706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7f5e0 msgr2=0x55e033d80800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7f5e0 0x55e033d80800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 -- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7d7f0 msgr2=0x55e033e17010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.369 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7d7f0 0x55e033e17010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient: _finish_hunting 0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 monclient: found mon.noname-c 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 -- 172.21.15.70:0/1930235706 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e033c79ca0 con 0x55e033d847c0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.261+0000 7fe3ac296640 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000a8b0 tx=0x7fe3a0005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b15f0226e78f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 1 -- 172.21.15.70:0/1930235706 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a001a070 con 0x55e033d847c0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: dump: 2024-09-16T03:50:57.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 1 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 3 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: _finish_auth 0 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: _check_auth_tickets 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 1 -- 172.21.15.70:0/1930235706 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a0008a50 con 0x55e033d847c0 2024-09-16T03:50:57.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 1 -- 172.21.15.70:0/1930235706 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0008e00 con 0x55e033d847c0 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aa292640 10 monclient: dump: 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 1 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 3 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:57.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:57.375 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.375 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.375 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.375 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_snapshots_pp 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.376 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_snapshots_pp' 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.377 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.378 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aec0f780 5 monclient: authenticate success, global_id 4379 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aec0f780 10 monclient: get_monmap_and_config success 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.262+0000 7fe3aec0f780 4 set_mon_vals no callback set 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.263+0000 7fe3aec0f780 10 monclient: shutdown 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.263+0000 7fe3aec0f780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.263+0000 7fe3aec0f780 1 -- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 msgr2=0x55e033d47b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.263+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000a8b0 tx=0x7fe3a0005ac0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 -- 172.21.15.70:0/1930235706 shutdown_connections 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033d47b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7f5e0 0x55e033d80800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/1930235706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7d7f0 0x55e033e17010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 -- 172.21.15.70:0/1930235706 >> 172.21.15.70:0/1930235706 conn(0x55e033d88010 msgr2=0x55e033e15a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:57.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 -- 172.21.15.70:0/1930235706 shutdown_connections 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 -- 172.21.15.70:0/1930235706 wait complete. 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 10 monclient: build_initial_monmap 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 10 monclient: monmap: 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 0 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 0.000000 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 0.000000 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 1 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 librados: starting msgr at 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 librados: starting objecter 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 Processor -- start 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 -- start start 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 librados: setting wanted keys 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 1 librados: calling monclient init 2024-09-16T03:50:57.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.264+0000 7fe3aec0f780 10 monclient: init 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.382 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient: _reopen_session rank -1 2024-09-16T03:50:57.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 0x55e033e0e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-b con 0x55e033d7d7f0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7f5e0 0x55e033e10710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-a con 0x55e033d7f5e0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033e12150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): picked mon.noname-c con 0x55e033d847c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e033d47340 con 0x55e033d7f5e0 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e033d47060 con 0x55e033d7d7f0 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x55e033d7f2f0 con 0x55e033d847c0 2024-09-16T03:50:57.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): _renew_subs 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.265+0000 7fe3aec0f780 10 monclient(hunting): authenticate will time out at 815.926594s 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 0x55e033e0e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 0x55e033e0e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.70:50026/0 (socket says 172.21.15.70:50026) 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 1 -- 172.21.15.70:0/2074496955 learned_addr learned my addr 172.21.15.70:0/2074496955 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): get_auth_request con 0x55e033d7d7f0 auth_method 0 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:57.385 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_snapshots_pp 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ab294640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7f5e0 0x55e033e10710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033e12150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ab294640 10 monclient(hunting): get_auth_request con 0x55e033d7f5e0 auth_method 0 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ab294640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ab294640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ab294640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): get_auth_request con 0x55e033d847c0 auth_method 0 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:57.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3aba95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.266+0000 7fe3ac296640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3ab294640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 -- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 msgr2=0x55e033e12150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033e12150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 -- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7f5e0 msgr2=0x55e033e10710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7f5e0 0x55e033e10710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:57.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 10 monclient: _finish_hunting 0 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39713/exe 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 monclient: found mon.noname-b 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e033e15330 con 0x55e033d7d7f0 2024-09-16T03:50:57.388 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3ac296640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x55e033d847c0 0x55e033e12150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3ab294640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e033d7f5e0 0x55e033e10710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3aba95640 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 0x55e033e0e4f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009430 tx=0x7fe3a40098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3a9290640 10 client.?.objecter ms_handle_connect 0x55e033d7d7f0 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.267+0000 7fe3a9290640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4014020 con 0x55e033d7d7f0 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: dump: 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 1 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.389 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 3 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: _finish_auth 0 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: _check_auth_tickets 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a4010920 con 0x55e033d7d7f0 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3aec0f780 5 monclient: authenticate success, global_id 4438 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4010d10 con 0x55e033d7d7f0 2024-09-16T03:50:57.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.393 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.394 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_snapshots_pp 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39847 2024-09-16T03:50:57.395 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_snapshots_pp on pid 39847' 2024-09-16T03:50:57.396 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640test api_snapshots_pp on pid 39847 2024-09-16T03:50:57.398 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39847 2024-09-16T03:50:57.398 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.398 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.399 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.400 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stdout: 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3a9290640 10 monclient: dump: 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: epoch 1 2024-09-16T03:50:57.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: created 2024-09-16T03:48:50.246472+0000 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: election_strategy: 3 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3aec0f780 10 monclient: _renew_subs 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3aec0f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e033e12690 con 0x55e033d7d7f0 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.268+0000 7fe3aa292640 4 set_mon_vals no callback set 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 10 monclient: _renew_subs 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e033eb28c0 con 0x55e033d7d7f0 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 1 librados: init done 2024-09-16T03:50:57.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.269+0000 7fe3aec0f780 10 librados: wait_for_osdmap waiting 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fe3a4017020 con 0x55e033d7d7f0 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.70:6804/198622594 conn(0x7fe39806be20 0x7fe39806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4118+0+0 (secure 0 0 0) 0x7fe3a4089830 con 0x55e033d7d7f0 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_map(22..22 src has 1..22) 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map got epochs [22,22] > 0 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.271+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map decoding full epoch 22 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3a9290640 20 client.4438.objecter dump_active .. 0 homeless 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3aec0f780 10 librados: wait_for_osdmap done waiting 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3aec0f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe3a0008a50 con 0x55e033d7d7f0 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.413 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.414 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.415 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_stat 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_stat' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.416 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.417 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_stat 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.418 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39650/exe 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.419 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39706/exe 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.420 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_stat 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39870 2024-09-16T03:50:57.421 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3ab294640 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.70:6804/198622594 conn(0x7fe39806be20 0x7fe39806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 test api_stat on pid 39870 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_stat on pid 39870' 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39870 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.434 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.435 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.436 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_stat_pp 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_stat_pp' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_stat_pp 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.437 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.438 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.439 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.440 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.441 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.442 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.443 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.444 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39771/exe 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_stat_pp 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39912 2024-09-16T03:50:57.445 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_stat_pp on pid 39912' 2024-09-16T03:50:57.446 INFO:tasks.workunit.client.0.smithi070.stdout:test api_stat_pp on pid 39912 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39912 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.454 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.455 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.456 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.457 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.458 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_watch_notify 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_watch_notify' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.459 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.460 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_watch_notify 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39819/exe 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.461 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_watch_notify 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39943 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_watch_notify on pid 39943' 2024-09-16T03:50:57.462 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.463 INFO:tasks.workunit.client.0.smithi070.stdout:test api_watch_notify on pid 39943 2024-09-16T03:50:57.474 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=39943 2024-09-16T03:50:57.474 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.474 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_watch_notify_pp' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_watch_notify_pp 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.475 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.476 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.477 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.478 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.479 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.480 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.481 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.482 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.483 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.484 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.485 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_watch_notify_pp 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40013 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_watch_notify_pp on pid 40013' 2024-09-16T03:50:57.486 INFO:tasks.workunit.client.0.smithi070.stdout:test api_watch_notify_pp on pid 40013 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40013 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.496 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39912/exe 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_cmd 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_cmd' 2024-09-16T03:50:57.497 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39847/exe 2024-09-16T03:50:57.498 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.499 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.500 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.501 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.502 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.503 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_cmd 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_cmd 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40072 2024-09-16T03:50:57.504 INFO:tasks.workunit.client.0.smithi070.stdout:test api_cmd on pid 40072 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_cmd on pid 40072' 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40072 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.514 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.515 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_cmd_pp 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.516 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.517 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_cmd_pp' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.518 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.519 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_cmd_pp 2024-09-16T03:50:57.520 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39870/exe 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.521 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.522 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.523 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/39943/exe 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_cmd_pp 2024-09-16T03:50:57.524 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40124 2024-09-16T03:50:57.525 INFO:tasks.workunit.client.0.smithi070.stdout:test api_cmd_pp on pid 40124 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_cmd_pp on pid 40124' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40124 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.529 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_service 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.530 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_service' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.531 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_service 2024-09-16T03:50:57.532 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.533 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_service 2024-09-16T03:50:57.533 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40145 2024-09-16T03:50:57.533 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_service on pid 40145' 2024-09-16T03:50:57.533 INFO:tasks.workunit.client.0.smithi070.stdout:test api_service on pid 40145 2024-09-16T03:50:57.537 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40145 2024-09-16T03:50:57.537 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.537 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_service_pp 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_service_pp' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.538 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_service_pp 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.539 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.540 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T03:50:57.541 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.542 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.542 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_service_pp 2024-09-16T03:50:57.542 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40157 2024-09-16T03:50:57.542 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_service_pp on pid 40157' 2024-09-16T03:50:57.542 INFO:tasks.workunit.client.0.smithi070.stdout:test api_service_pp on pid 40157 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40157 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.550 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.551 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_c_write_operations 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.552 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_c_write_operations' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.553 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_c_write_operations 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40072/exe 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.554 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.555 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40013/exe 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.556 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_c_write_operations 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40181 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_c_write_operations on pid 40181' 2024-09-16T03:50:57.557 INFO:tasks.workunit.client.0.smithi070.stdout:test api_c_write_operations on pid 40181 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40181 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.565 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.566 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.567 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.568 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_c_read_operations 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_c_read_operations' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.569 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.570 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.571 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_c_read_operations 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_c_read_operations 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40213 2024-09-16T03:50:57.572 INFO:tasks.workunit.client.0.smithi070.stdout:test api_c_read_operations on pid 40213 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_c_read_operations on pid 40213' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40213 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.581 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s api_cls_remote_reads 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' api_cls_remote_reads' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.582 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.583 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo api_cls_remote_reads 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.584 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.585 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.586 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.587 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=api_cls_remote_reads 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40239 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test api_cls_remote_reads on pid 40239' 2024-09-16T03:50:57.588 INFO:tasks.workunit.client.0.smithi070.stdout:test api_cls_remote_reads on pid 40239 2024-09-16T03:50:57.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40239 2024-09-16T03:50:57.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s list_parallel 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.602 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' list_parallel' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.604 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.605 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo list_parallel 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.606 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.607 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.608 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.609 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.610 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40145/exe 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.611 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout:comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3ab294640 10 monclient: get_auth_request con 0x7fe39806be20 auth_method 0 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe3a4054210 con 0x55e033d7d7f0 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3a9290640 10 monclient: handle_get_version_reply finishing 1 version 22 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.272+0000 7fe3a9a91640 10 client.4438.objecter _get_latest_version latest 22, have it 2024-09-16T03:50:57.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3aec0f780 10 client.4438.objecter create_pool name=ceph_test_rados_api_asio 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3aec0f780 10 client.4438.objecter pool_op_submit 1 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3aec0f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55e033cfc590 con 0x55e033d7d7f0 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3ab294640 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.70:6804/198622594 conn(0x7fe39806be20 0x7fe39806e2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe39c057000 tx=0x7fe39c051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.273+0000 7fe3a9290640 10 client.4438.objecter ms_handle_connect 0x7fe39806be20 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v23) ==== 43+0+0 (secure 0 0 0) 0x7fe3a401e030 con 0x55e033d7d7f0 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 pool_op_reply(tid 1 (0) Success v23) 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter have request 1 at 0x55e033e15500 Op: create 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 20 client.4438.objecter waiting for client to reach epoch 23 before calling back 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 monclient: _renew_subs 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.163:3300/0 2024-09-16T03:50:57.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=23}) -- 0x7fe398071780 con 0x55e033d7d7f0 2024-09-16T03:50:57.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter done 2024-09-16T03:50:57.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== mon.1 v2:172.21.15.163:3300/0 8 ==== osd_map(23..23 src has 1..23) ==== 2488+0+0 (secure 0 0 0) 0x7fe3a4056010 con 0x55e033d7d7f0 2024-09-16T03:50:57.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_map(23..23 src has 1..23) 2024-09-16T03:50:57.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map got epochs [23,23] > 22 2024-09-16T03:50:57.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map decoding incremental epoch 23 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.623 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.624 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.625 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.626 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.627 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.628 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40157/exe 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.629 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_service: /' 2024-09-16T03:50:57.630 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.631 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.632 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.633 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=list_parallel 2024-09-16T03:50:57.634 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40315 2024-09-16T03:50:57.634 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test list_parallel on pid 40315' 2024-09-16T03:50:57.634 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3a9290640 20 client.4438.test list_parallel on pid 40315 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40315 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s open_pools_parallel 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.639 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' open_pools_parallel' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.640 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40181/exe 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.641 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.642 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo open_pools_parallel 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=open_pools_parallel 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.643 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40327 2024-09-16T03:50:57.644 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test open_pools_parallel on pid 40327' 2024-09-16T03:50:57.644 INFO:tasks.workunit.client.0.smithi070.stdout:test open_pools_parallel on pid 40327 2024-09-16T03:50:57.651 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40327 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40124/exe 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.652 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.653 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.654 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.655 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.656 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.657 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s delete_pools_parallel 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' delete_pools_parallel' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo delete_pools_parallel 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.658 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=delete_pools_parallel 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.659 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40362 2024-09-16T03:50:57.660 INFO:tasks.workunit.client.0.smithi070.stdout:test delete_pools_parallel on pid 40362 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test delete_pools_parallel on pid 40362' 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40362 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.670 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.671 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40239/exe 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.672 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.673 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.674 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.675 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.676 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s cls 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.677 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' cls' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.678 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo cls 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.679 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40213/exe 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=cls 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40394 2024-09-16T03:50:57.680 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test cls on pid 40394' 2024-09-16T03:50:57.681 INFO:tasks.workunit.client.0.smithi070.stdout:test cls on pid 40394 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40394 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s cmd 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.691 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' cmd' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.692 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.693 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.694 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.695 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.696 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.697 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo cmd 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.698 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.699 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.700 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=cmd 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40327/exe 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40424 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test cmd on pid 40424' 2024-09-16T03:50:57.701 INFO:tasks.workunit.client.0.smithi070.stdout:test cmd on pid 40424 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40424 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s handler_error 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' handler_error' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.709 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.710 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.711 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.712 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.713 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.714 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo handler_error 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=handler_error 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40450 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test handler_error on pid 40450' 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-16T03:50:57.715 INFO:tasks.workunit.client.0.smithi070.stdout:test handler_error on pid 40450 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40450 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s io 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' io' 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.720 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.721 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.722 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo io 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=io 2024-09-16T03:50:57.723 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40467 2024-09-16T03:50:57.724 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test io on pid 40467' 2024-09-16T03:50:57.724 INFO:tasks.workunit.client.0.smithi070.stdout:test io on pid 40467 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40467 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.734 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s ec_io 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' ec_io' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.735 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.736 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.737 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.738 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.739 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.740 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.741 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40362/exe 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.742 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo ec_io 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.743 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=ec_io 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40513 2024-09-16T03:50:57.744 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test ec_io on pid 40513' 2024-09-16T03:50:57.745 INFO:tasks.workunit.client.0.smithi070.stdout:test ec_io on pid 40513 2024-09-16T03:50:57.759 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40513 2024-09-16T03:50:57.759 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.759 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.759 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s list 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' list' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo list 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.760 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.761 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.762 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.763 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40315/exe 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.764 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.765 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.766 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.767 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.768 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40394/exe 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.769 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-16T03:50:57.770 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_cls 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40424/exe 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.771 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.772 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=list 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40570 2024-09-16T03:50:57.773 INFO:tasks.workunit.client.0.smithi070.stdout:test list on pid 40570 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test list on pid 40570' 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40570 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.784 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.785 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_cmd 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.786 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ cls: /' 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_cls.log 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s ec_list 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' ec_list' 2024-09-16T03:50:57.787 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ cmd: /' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.788 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.789 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.790 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.791 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.792 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo ec_list 2024-09-16T03:50:57.793 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=ec_list 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.794 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40624 2024-09-16T03:50:57.795 INFO:tasks.workunit.client.0.smithi070.stdout:test ec_list on pid 40624 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test ec_list on pid 40624' 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40624 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.802 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.803 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40450/exe 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.804 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s misc 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' misc' 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40467/exe 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ handler_error: /' 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.805 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.806 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ io: /' 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_io.log 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_io 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.807 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_handler_error 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo misc 2024-09-16T03:50:57.808 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=misc 2024-09-16T03:50:57.809 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40697 2024-09-16T03:50:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:test misc on pid 40697 2024-09-16T03:50:57.817 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test misc on pid 40697' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40697 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.818 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.819 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.820 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s pool 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' pool' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo pool 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.821 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=pool 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40768 2024-09-16T03:50:57.822 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test pool on pid 40768' 2024-09-16T03:50:57.823 INFO:tasks.workunit.client.0.smithi070.stdout:test pool on pid 40768 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40768 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s read_operations 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' read_operations' 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.829 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.830 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo read_operations 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.831 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.832 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=read_operations 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40798 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test read_operations on pid 40798' 2024-09-16T03:50:57.833 INFO:tasks.workunit.client.0.smithi070.stdout:test read_operations on pid 40798 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40798 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s snapshots 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' snapshots' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.842 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.843 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.844 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo snapshots 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.845 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.846 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.847 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.848 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=snapshots 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40829 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test snapshots on pid 40829' 2024-09-16T03:50:57.849 INFO:tasks.workunit.client.0.smithi070.stdout:test snapshots on pid 40829 2024-09-16T03:50:57.865 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40829 2024-09-16T03:50:57.865 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s watch_notify 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' watch_notify' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.866 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.867 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.868 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.869 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40513/exe 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.870 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.871 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.872 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.873 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo watch_notify 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.874 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ ec_io: /' 2024-09-16T03:50:57.875 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.876 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.877 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-16T03:50:57.878 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_ec_io 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40570/exe 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.879 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.880 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_list.log 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_list 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.881 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=watch_notify 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40891 2024-09-16T03:50:57.882 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test watch_notify on pid 40891' 2024-09-16T03:50:57.883 INFO:tasks.workunit.client.0.smithi070.stdout:test watch_notify on pid 40891 2024-09-16T03:50:57.891 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40891 2024-09-16T03:50:57.891 INFO:tasks.workunit.client.0.smithi070.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:50:57.891 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40624/exe 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:++ printf %25s write_operations 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+ r=' write_operations' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ list: /' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.892 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++ echo write_operations 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++ awk '{print $1}' 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.893 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.894 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.895 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.896 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.897 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.898 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.899 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ ff=write_operations 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ ec_list: /' 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40925 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stderr:+ echo 'test write_operations on pid 40925' 2024-09-16T03:50:57.900 INFO:tasks.workunit.client.0.smithi070.stdout:test write_operations on pid 40925 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ pids[$f]=40925 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ ret=0 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39713 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39713 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40697/exe 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.928 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.929 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.930 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.931 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40768/exe 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_misc 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.932 INFO:tasks.workunit.client.0.smithi070.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_pool 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_pool.log 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.933 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -f /etc/bashrc ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+ . /etc/bashrc 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -z '' ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ BASHRCSOURCED=Y 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:++ shopt -q login_shell 2024-09-16T03:50:57.934 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ pool: /' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:+++ umask 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' 0022 -eq 0 ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ SHELL=/bin/bash 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.935 INFO:tasks.workunit.client.0.smithi070.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_misc.log 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.936 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ misc: /' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_ec_list 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.937 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.938 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.939 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'grep=grep --color=auto' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorls.sh 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' '!' -t 0 ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:+++ return 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-16T03:50:57.940 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export S_COLORS=auto 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ S_COLORS=auto 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.941 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.942 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.943 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.944 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.945 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.946 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40829/exe 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40798/exe 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.947 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_snapshots 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.948 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_read_operations 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40891/exe 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.949 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ read_operations: /' 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ snapshots: /' 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_watch_notify 2024-09-16T03:50:57.950 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/gawk.sh 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-16T03:50:57.951 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/lang.sh 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/locale 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ watch_notify: /' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:++++ export LANG=en_US.UTF-8 2024-09-16T03:50:57.952 INFO:tasks.workunit.client.0.smithi070.stderr:++++ LANG=en_US.UTF-8 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ LANG=en_US.UTF-8 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ unset LANG_backup config 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n '' ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' dumb = linux ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/less.sh 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-16T03:50:57.953 INFO:tasks.workunit.client.0.smithi070.stderr:++ for i in /etc/profile.d/*.sh 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++ '[' '' ']' 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++ . /etc/profile.d/which2.sh 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++++ readlink /proc/40925/exe 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++++ basename /usr/bin/bash 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_declare='declare -f' 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++ which_opt=-f 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export which_declare 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+++ export -f which 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset i 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:++ unset -f pathmunge 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-16T03:50:57.954 INFO:tasks.workunit.client.0.smithi070.stderr:+ export PATH 2024-09-16T03:50:57.955 INFO:tasks.workunit.client.0.smithi070.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-16T03:50:57.955 INFO:tasks.workunit.client.0.smithi070.stderr:+ unset rc 2024-09-16T03:50:57.955 INFO:tasks.workunit.client.0.smithi070.stderr:+ ceph_test_neorados_write_operations 2024-09-16T03:50:57.955 INFO:tasks.workunit.client.0.smithi070.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-16T03:50:57.955 INFO:tasks.workunit.client.0.smithi070.stderr:+ sed 's/^/ write_operations: /' 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout:objecter dump_active .. 0 homeless 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 10 librados: set snap read head -> 1 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 10 librados: writefull oid=exist nspace= 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e08ee0 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.163:6804/1904387775 conn(0x55e033d45f40 0x55e033e01be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:58.483 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 2 osd.4 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 2 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 15 client.4438.objecter _send_op 2 to 5.1 on osd.4 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:2 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x55e033e02120 con 0x55e033d45f40 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.437+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.451+0000 7fe3ac296640 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.163:6804/1904387775 conn(0x55e033d45f40 0x55e033e01be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.451+0000 7fe3ac296640 10 monclient: get_auth_request con 0x55e033d45f40 auth_method 0 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.457+0000 7fe3ac296640 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.163:6804/1904387775 conn(0x55e033d45f40 0x55e033e01be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:57.458+0000 7fe3a9290640 10 client.4438.objecter ms_handle_connect 0x55e033d45f40 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 1 ==== osd_map(24..24 src has 1..24) ==== 13784+0+0 (crc 0 0 0) 0x7fe3a0008850 con 0x55e033d45f40 2024-09-16T03:50:58.484 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_map(24..24 src has 1..24) 2024-09-16T03:50:58.485 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map got epochs [24,24] > 23 2024-09-16T03:50:58.485 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 3 client.4438.objecter handle_osd_map decoding incremental epoch 24 2024-09-16T03:50:58.485 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 10 client.4438.objecter checking op 2 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 20 client.4438.objecter dump_active .. 0 homeless 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.481+0000 7fe3a9290640 20 client.4438.objecter 2 5.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 2 ==== osd_op_reply(2 exist [writefull 0~5] v24'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.575 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(2 exist [writefull 0~5] v24'1 uv1 ondisk = 0) 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 2 ondisk uv 1 in 5.1 attempt 0 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 15 client.4438.objecter _finish_op 2 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 2 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 librados: Objecter returned from writefull r=0 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e08ee0 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.576 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 3 osd.4 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 3 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 15 client.4438.objecter _send_op 3 to 5.1 on osd.4 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:3 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033e024c0 con 0x55e033d45f40 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.577 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 4 osd.4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 15 client.4438.objecter _send_op 4 to 5.1 on osd.4 2024-09-16T03:50:58.578 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:4 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033e12e00 con 0x55e033d45f40 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.573+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (3 ms) 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 3 ondisk uv 1 in 5.1 attempt 0 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter _finish_op 3 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.579 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 3 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 4 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 4 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter _finish_op 4 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 4 2024-09-16T03:50:58.580 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 5 osd.4 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 5 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 15 client.4438.objecter _send_op 5 to 5.1 on osd.4 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:5 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033e08ee0 con 0x55e033d45f40 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e131f0 2024-09-16T03:50:58.581 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 6 osd.4 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 6 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 15 client.4438.objecter _send_op 6 to 5.1 on osd.4 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:6 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec5bc0 con 0x55e033d45f40 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.575+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.582 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 5 ondisk uv 1 in 5.1 attempt 0 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter _finish_op 5 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 5 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 6 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.583 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 6 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter _finish_op 6 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 6 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.584 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 7 osd.4 2024-09-16T03:50:58.585 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.585 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 7 2024-09-16T03:50:58.585 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 15 client.4438.objecter _send_op 7 to 5.1 on osd.4 2024-09-16T03:50:58.585 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:7 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ee5f70 con 0x55e033d45f40 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033f06320 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 8 osd.4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 8 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 15 client.4438.objecter _send_op 8 to 5.1 on osd.4 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:8 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033f067d0 con 0x55e033d45f40 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.586 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.576+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 7 ondisk uv 1 in 5.1 attempt 0 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter _finish_op 7 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 7 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.587 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 8 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 8 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter _finish_op 8 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 8 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033c92850 write_seq 1 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.588 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 9 osd.4 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 9 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 15 client.4438.objecter _send_op 9 to 5.1 on osd.4 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:9 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033e131f0 con 0x55e033d45f40 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.577+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 9 ==== osd_op_reply(9 exist [write 0~5] v24'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(9 exist [write 0~5] v24'2 uv2 ondisk = 0) 2024-09-16T03:50:58.589 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 9 ondisk uv 2 in 5.1 attempt 0 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 9 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 15 client.4438.objecter _finish_op 9 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 9 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033c92850 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (3 ms) 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033cb9d40 write_seq 2 2024-09-16T03:50:58.590 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 10 osd.4 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 10 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 15 client.4438.objecter _send_op 10 to 5.1 on osd.4 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:10 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033e131f0 con 0x55e033d45f40 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.581+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 10 ==== osd_op_reply(10 exist [write 0~5] v24'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.591 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(10 exist [write 0~5] v24'3 uv3 ondisk = 0) 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 10 ondisk uv 3 in 5.1 attempt 0 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 10 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 15 client.4438.objecter _finish_op 10 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 10 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033cb9d40 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033f06320 write_seq 3 2024-09-16T03:50:58.592 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 11 osd.4 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 11 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 15 client.4438.objecter _send_op 11 to 5.1 on osd.4 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:11 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec9560 con 0x55e033d45f40 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.582+0000 7fe3aec0f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 11 ==== osd_op_reply(11 exist [write 0~5] v24'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(11 exist [write 0~5] v24'4 uv4 ondisk = 0) 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 11 ondisk uv 4 in 5.1 attempt 0 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 11 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 15 client.4438.objecter _finish_op 11 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 11 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.583+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033f06320 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 12 osd.4 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 12 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _send_op 12 to 5.1 on osd.4 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:12 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec8ef0 con 0x55e033d45f40 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033ec9290 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 13 osd.4 2024-09-16T03:50:58.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 13 2024-09-16T03:50:58.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _send_op 13 to 5.1 on osd.4 2024-09-16T03:50:58.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:13 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec5f60 con 0x55e033d45f40 2024-09-16T03:50:58.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 12 ondisk uv 4 in 5.1 attempt 0 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 12 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _finish_op 12 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 12 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 13 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 13 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _finish_op 13 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 13 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 14 osd.4 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 14 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _send_op 14 to 5.1 on osd.4 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:14 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec8ef0 con 0x55e033d45f40 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033ec5f60 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 15 osd.4 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 15 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 15 client.4438.objecter _send_op 15 to 5.1 on osd.4 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:15 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec94f0 con 0x55e033d45f40 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 14 ondisk uv 4 in 5.1 attempt 0 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 14 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _finish_op 14 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 14 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.584+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 15 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 15 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _finish_op 15 2024-09-16T03:50:58.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 15 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 16 osd.4 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 16 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _send_op 16 to 5.1 on osd.4 2024-09-16T03:50:58.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:16 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec8ef0 con 0x55e033d45f40 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033ec65d0 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 17 osd.4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 17 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _send_op 17 to 5.1 on osd.4 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:17 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec6a80 con 0x55e033d45f40 2024-09-16T03:50:58.604 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 5 client.4438.objecter 2 in flight 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 16 ondisk uv 4 in 5.1 attempt 0 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 5 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 16 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _finish_op 16 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 5 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 16 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.605 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 17 ondisk uv 0 in 5.1 attempt 0 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 17 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _finish_op 17 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 17 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033f065a0 write_seq 4 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.606 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 18 osd.4 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 18 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 15 client.4438.objecter _send_op 18 to 5.1 on osd.4 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:18 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec8ef0 con 0x55e033d45f40 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.585+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 18 ==== osd_op_reply(18 exist [writefull 0~5] v24'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(18 exist [writefull 0~5] v24'5 uv5 ondisk = 0) 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.607 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 18 ondisk uv 5 in 5.1 attempt 0 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 18 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 15 client.4438.objecter _finish_op 18 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 18 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033f065a0 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033f06490 write_seq 5 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.608 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 19 osd.4 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 19 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 15 client.4438.objecter _send_op 19 to 5.1 on osd.4 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:19 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033ec8ef0 con 0x55e033d45f40 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.586+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 19 ==== osd_op_reply(19 exist [writefull 0~5] v24'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(19 exist [writefull 0~5] v24'6 uv6 ondisk = 0) 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.609 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 19 ondisk uv 6 in 5.1 attempt 0 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 19 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 15 client.4438.objecter _finish_op 19 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 19 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033f06490 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 librados: queue_aio_write 0x55e033cc2a10 completion 0x55e033ec8ef0 write_seq 6 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 10 client.4438.objecter _op_submit op 0x55e033e12950 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 client.4438.objecter _calc_target epoch 24 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 client.4438.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-16T03:50:58.610 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 10 client.4438.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [4,3] primary 4 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 client.4438.objecter _get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 10 client.4438.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 20 osd.4 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 client.4438.objecter get_session s=0x55e033d46a70 osd=4 3 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 15 client.4438.objecter _session_op_assign 4 20 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 15 client.4438.objecter _send_op 20 to 5.1 on osd.4 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 --> v2:172.21.15.163:6804/1904387775 -- osd_op(unknown.0.0:20 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x55e033eca000 con 0x55e033d45f40 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.587+0000 7fe3aec0f780 5 client.4438.objecter 1 in flight 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 <== osd.4 v2:172.21.15.163:6804/1904387775 20 ==== osd_op_reply(20 exist [writefull 0~5] v24'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fe3a001d040 con 0x55e033d45f40 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 10 client.4438.objecter ms_dispatch 0x55e033ccd700 osd_op_reply(20 exist [writefull 0~5] v24'7 uv7 ondisk = 0) 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 10 client.4438.objecter in handle_osd_op_reply 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 7 client.4438.objecter handle_osd_op_reply 20 ondisk uv 7 in 5.1 attempt 0 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 10 client.4438.objecter op 0 rval 0 len 0 2024-09-16T03:50:58.611 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 15 client.4438.objecter handle_osd_op_reply completed tid 20 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 15 client.4438.objecter _finish_op 20 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 4 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 15 client.4438.objecter _session_op_remove 4 20 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 5 client.4438.objecter 0 in flight 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3ac296640 20 librados: complete_aio_write 0x55e033ec8ef0 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 10 librados: watch_flush enter 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 10 librados: watch_flush exit 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 10 client.4438.objecter close_session for osd.4 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 >> v2:172.21.15.163:6804/1904387775 conn(0x55e033d45f40 msgr2=0x55e033e01be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.163:6804/1904387775 conn(0x55e033d45f40 0x55e033e01be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 20 client.4438.objecter put_session s=0x55e033d46a70 osd=4 1 2024-09-16T03:50:58.612 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 20 client.4438.objecter shutdown clearing up homeless session... 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 10 client.4438.objecter successfully canceled tick 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 >> v2:172.21.15.70:6804/198622594 conn(0x7fe39806be20 msgr2=0x7fe39806e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/2074496955 >> v2:172.21.15.70:6804/198622594 conn(0x7fe39806be20 0x7fe39806e2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe39c057000 tx=0x7fe39c051000 comp rx=0 tx=0).stop 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 10 monclient: shutdown 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.588+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 msgr2=0x55e033e0e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 --2- 172.21.15.70:0/2074496955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e033d7d7f0 0x55e033e0e4f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009430 tx=0x7fe3a40098f0 comp rx=0 tx=0).stop 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3ac296640 1 -- 172.21.15.70:0/2074496955 reap_dead start 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 shutdown_connections 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 >> 172.21.15.70:0/2074496955 conn(0x55e033d88010 msgr2=0x55e033d47990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 shutdown_connections 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 -- 172.21.15.70:0/2074496955 wait complete. 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.589+0000 7fe3aec0f780 1 librados: shutdown 2024-09-16T03:50:58.613 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [----------] 12 tests from AsioRados (16 ms total) 2024-09-16T03:50:58.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: 2024-09-16T03:50:58.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T03:50:58.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1330 ms total) 2024-09-16T03:50:58.614 INFO:tasks.workunit.client.0.smithi070.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T03:50:58.808 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.ListObjects (110 ms) 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T03:50:58.809 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (19 ms) 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo1 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo2 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo3 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:50:58.810 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo4 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo5 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo1 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo6,foo7 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo7 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo6 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:50:58.811 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo4 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo5 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns2:foo7 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns2:foo6 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo1 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo1 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo2 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo3 2024-09-16T03:50:58.812 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (60 ms) 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 1 0 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 10 0 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 13 0 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 7 0 2024-09-16T03:50:58.813 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 14 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 0 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 15 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 11 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 5 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 8 0 2024-09-16T03:50:58.814 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 6 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 3 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 4 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 12 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 9 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 2 0 2024-09-16T03:50:58.815 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (59 ms) 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: x cursor=MIN 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-09-16T03:50:58.816 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-16T03:50:58.817 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-16T03:50:58.818 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-09-16T03:50:58.863 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-09-16T03:50:58.864 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-09-16T03:50:58.865 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=10 expected=10 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=13 expected=13 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-16T03:50:58.866 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=1 expected=1 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=12 expected=12 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-09-16T03:50:58.867 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=4 expected=4 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=3 expected=3 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=2 expected=2 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-16T03:50:58.868 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=9 expected=9 2024-09-16T03:50:58.869 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-16T03:51:00.486 INFO:tasks.workunit.client.0.smithi070.stdout: api_ api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-16T03:51:00.486 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (2913 ms) 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (2913 ms total) 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (2913 ms total) 2024-09-16T03:51:00.487 INFO:tasks.workunit.client.0.smithi070.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (52 ms) 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (86 ms) 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2874 ms) 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3014 ms total) 2024-09-16T03:51:00.523 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: 2024-09-16T03:51:00.524 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T03:51:00.524 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3014 ms total) 2024-09-16T03:51:00.524 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (279 ms) 2024-09-16T03:51:00.549 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (15 ms) 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (17 ms) 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (24 ms) 2024-09-16T03:51:00.550 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (4 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (13 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (11 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (7 ms) 2024-09-16T03:51:00.551 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (6 ms) 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (26 ms) 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (28 ms) 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T03:51:00.552 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (31 ms) 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (3 ms) 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (478 ms total) 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3009 ms total) 2024-09-16T03:51:00.553 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T03:51:00.554 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: Running main() from gmock_main.cc 2024-09-16T03:51:00.554 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-16T03:51:00.554 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [----------] Global test environment set-up. 2024-09-16T03:51:00.554 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2853 ms) 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [----------] 1 test from neocls_handler_error (2853 ms total) 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [----------] Global test environment tear-down 2024-09-16T03:51:00.555 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2854 ms total) 2024-09-16T03:51:00.556 INFO:tasks.workunit.client.0.smithi070.stdout: handler_error: [ PASSED ] 1 test. 2024-09-16T03:51:00.820 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: starting. 2024-09-16T03:51:00.820 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: creating pool ceph_test_rados_list_parallel.smithi070-40537 2024-09-16T03:51:00.820 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: created object 0... 2024-09-16T03:51:00.820 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: created object 25... 2024-09-16T03:51:00.820 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: created object 49... 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: finishing. 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_1_[40584]: shutting down. 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: starting. 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: listing objects. 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: listed object 0... 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: listed object 25... 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: saw 50 objects 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_2_[40585]: shutting down. 2024-09-16T03:51:00.821 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: starting. 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: creating pool ceph_test_rados_list_parallel.smithi070-40537 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: created object 0... 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: created object 25... 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: created object 49... 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: finishing. 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_3_[41340]: shutting down. 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: starting. 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: removed 25 objects... 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: removed half of the objects 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: removed 50 objects... 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: removed 50 objects 2024-09-16T03:51:00.822 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_5_[41342]: shutting down. 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_4_[41341]: starting. 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_4_[41341]: listing objects. 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_4_[41341]: listed object 0... 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_4_[41341]: saw 8 objects 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_4_[41341]: shutting down. 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: starting. 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: creating pool ceph_test_rados_list_parallel.smithi070-40537 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: created object 0... 2024-09-16T03:51:00.823 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: created object 25... 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: created object 49... 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: finishing. 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_6_[41391]: shutting down. 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: starting. 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: listing objects. 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: listed object 0... 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: listed object 25... 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: listed object 50... 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: saw 54 objects 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_7_[41392]: shutting down. 2024-09-16T03:51:00.824 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: starting. 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: added 25 objects... 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: added half of the objects 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: added 50 objects... 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: added 50 objects 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_8_[41393]: shutting down. 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:00.825 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: starting. 2024-09-16T03:51:00.836 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (235 ms) 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (49 ms) 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2809 ms) 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.443930+0000 mon.a [INF] from='client.? 172.21.15.70:0/1790957904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi070-40136-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.837 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444002+0000 mon.a [INF] from='client.? 172.21.15.70:0/3387964816' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi070-40116-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444062+0000 mon.a [INF] from='client.? 172.21.15.70:0/1157348232' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi070-39646-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444130+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40383-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444212+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi070-39848-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444285+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40178-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi070-39946-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444412+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi070-40399-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444460+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444513+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444555+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi070-39872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:00.838 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444636+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi070-39664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444701+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application en list_parallel: process_9_[41442]: creating pool ceph_test_rados_list_parallel.smithi070-40537 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: created object 0... 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: created object 25... 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: created object 49... 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: finishing. 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_9_[41442]: shutting down. 2024-09-16T03:51:01.121 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: starting. 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: listing objects. 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: listed object 0... 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: listed object 25... 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: listed object 50... 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: listed object 75... 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: saw 97 objects 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_10_[41443]: shutting down. 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.122 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: starting. 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: removed 25 objects... 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: removed half of the objects 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: removed 50 objects... 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: removed 50 objects 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_13_[41446]: shutting down. 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: starting. 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: added 25 objects... 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: added half of the objects 2024-09-16T03:51:01.123 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: added 50 objects... 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: added 50 objects 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_12_[41445]: shutting down. 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: starting. 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: added 25 objects... 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: added half of the objects 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: added 50 objects... 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: added 50 objects 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_11_[41444]: shutting down. 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.124 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: starting. 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: creating pool ceph_test_rados_list_parallel.smithi070-40537 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: created object 0... 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: created object 25... 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: created object 49... 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: finishing. 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_14_[41792]: shutting down. 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.125 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: starting. 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listing objects. 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 0... 2024-09-16T03:51:01.126 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 25... 2024-09-16T03:51:01.526 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 50... 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 75... 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 100... 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: listed object 125... 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: saw 150 objects 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_15_[41793]: shutting down. 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: starting. 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: added 25 objects... 2024-09-16T03:51:01.527 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: added half of the objects 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: added 50 objects... 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: added 50 objects 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: process_16_[41794]: shutting down. 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******************************* 2024-09-16T03:51:01.528 INFO:tasks.workunit.client.0.smithi070.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout:able","pool": "LibRadosStatPP_smithi070-40031-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40343-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.444843+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi070-39693-1","var":"pg_num","val":"11"}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.458188+0000 mon.c [INF] from='client.? 172.21.15.70:0/2338524872' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi070-39693-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.537308+0000 mon.a [WRN] Health check failed: 30 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:50:59.540669+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi070-39693-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.448986+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi070-40863-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi070-40863-1"}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.449086+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi070-40921-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi070-40921-1"}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.449165+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi070-40183-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi070-40183-1"}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.449236+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi070-39693-1","var":"pgp_num","val":"11"}]': finished 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.477946+0000 mon.b [INF] from='client.? 172.21.15.70:0/657650792' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39848-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.615 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.479291+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39848-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.483790+0000 mon.a [INF] from='client.? 172.21.15.70:0/3301968571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi070-39615-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.484068+0000 mon.c [INF] from='client.? 172.21.15.70:0/2014474837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi070-39637-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.484476+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi070-39637-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.667667+0000 mon.c [INF] from='client.? 172.21.15.70:0/2423522876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.668136+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.670777+0000 mon.c [INF] from='client.? 172.21.15.70:0/2423522876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.671402+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.672180+0000 mon.c [INF] from='client.? 172.21.15.70:0/2423522876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.672757+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.727252+0000 mon.b [INF] from='client.? 172.21.15.70:0/2227015268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.730277+0000 mon.b [INF] from='client.? 172.21.15.70:0/1842986467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.785545+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.786870+0000 mon.b [INF] from='client.? 172.21.15.70:0/2227015268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.795703+0000 mon.c [INF] from='client.? 172.21.15.70:0/1029088860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.616 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.808856+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.809413+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.809577+0000 mon.b [INF] from='client.? 172.21.15.70:0/1842986467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.809688+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.810006+0000 client.admin [INF] onexx 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.810515+0000 mon.c [INF] from='client.? 172.21.15.70:0/1029088860' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.810525+0000 mon.b [INF] from='client.? 172.21.15.70:0/2227015268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.832918+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.833198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.833805+0000 mon.b [INF] from='client.? 172.21.15.70:0/1842986467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi070-40116-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.833993+0000 mon.c [INF] from='client.? 172.21.15.70:0/1029088860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi070-39646-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.834137+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.834612+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi070-40116-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.617 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.835015+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi070-39646-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:00.835216+0000 mon.a [INF] from='client.? 172.21.15.70:0/295175824' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453090+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39848-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453220+0000 mon.a [INF] from='client.? 172.21.15.70:0/3301968571' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi070-39615-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453307+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi070-39637-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453446+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453531+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi070-40116-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.453619+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi070-39646-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.457902+0000 mon.c [INF] from='client.? 172.21.15.70:0/1029088860' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi070-39646-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.461308+0000 mon.c [INF] from='client.? 172.21.15.70:0/2423522876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi070-39664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.462089+0000 mon.b [INF] from='client.? 172.21.15.70:0/1842986467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi070-40116-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.465516+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi070-39646-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:01.618 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.466218+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi070-39664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.466550+0000 mon.c [INF] from='client.? 172.21.15.70:0/3179252632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.467363+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.467657+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi070-40116-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.469816+0000 mon.b [INF] from='client.? 172.21.15.70:0/438457596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi070-40863-1"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.470329+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi070-40863-1"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.471497+0000 mon.b [INF] from='client.? 172.21.15.70:0/2227015268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi070-40031-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.473185+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi070-40031-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:01.619 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.474503+0000 mon.b [INF] from='client.? 172.21.15.70:0/2339656452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.483442+0000 mon.c [INF] from='client.4774 172.21.15.70:0/2339656452' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-2","app": "rados","yes_i_re cls: Running main() from gmock_main.cc 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [----------] Global test environment set-up. 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-16T03:51:03.537 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [ OK ] NeoRadosCls.DNE (2901 ms) 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (3047 ms) 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [----------] 2 tests from NeoRadosCls (5948 ms total) 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [----------] Global test environment tear-down 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [==========] 2 tests from 1 test suite ran. (5948 ms total) 2024-09-16T03:51:03.538 INFO:tasks.workunit.client.0.smithi070.stdout: cls: [ PASSED ] 2 tests. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: starting. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: creating pool ceph_test_rados_open_pools_parallel.smithi070-40429 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: created object 0... 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: created object 25... 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: created object 49... 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: finishing. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_1_[40498]: shutting down. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_2_[40499]: starting. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_2_[40499]: rados_pool_create. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_2_[40499]: rados_ioctx_create. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_2_[40499]: shutting down. 2024-09-16T03:51:03.541 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: ******************************* 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: starting. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: creating pool ceph_test_rados_open_pools_parallel.smithi070-40429 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: created object 0... 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: created object 25... 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: created object 49... 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: finishing. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_3_[41458]: shutting down. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: ******************************* 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_4_[41459]: starting. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_4_[41459]: rados_pool_create. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_4_[41459]: rados_ioctx_create. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: process_4_[41459]: shutting down. 2024-09-16T03:51:03.542 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: ******************************* 2024-09-16T03:51:03.543 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: ******************************* 2024-09-16T03:51:03.543 INFO:tasks.workunit.client.0.smithi070.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: starting. 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: creating pool ceph_test_rados_delete_pools_parallel.smithi070-40528 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: created object 0... 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: created object 25... 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: created object 49... 2024-09-16T03:51:03.547 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: finishing. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_1_[40579]: shutting down. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_2_[40580]: starting. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_2_[40580]: deleting pool ceph_test_rados_delete_pools_parallel.smithi070-40528 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_2_[40580]: shutting down. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: starting. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: creating pool ceph_test_rados_delete_pools_parallel.smithi070-40528 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: created object 0... 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: created object 25... 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: created object 49... 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: finishing. 2024-09-16T03:51:03.548 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_3_[41455]: shutting down. 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: starting. 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: listing objects. 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: listed object 0... 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: listed object 25... 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: saw 50 objects 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_5_[41457]: shutting down. 2024-09-16T03:51:03.549 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_4_[41456]: starting. 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_4_[41456]: deleting pool ceph_test_rados_delete_pools_parallel.smithi070-40528 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: process_4_[41456]: shutting down. 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:51:03.552 INFO:tasks.workunit.client.0.smithi070.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:51:04.514 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: Running main() from gmock_main.cc 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [----------] Global test environment set-up. 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1966 ms) 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1923 ms) 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3029 ms) 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6918 ms total) 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [----------] Global test environment tear-down 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6920 ms total) 2024-09-16T03:51:04.515 INFO:tasks.workunit.client.0.smithi070.stdout: cmd: [ PASSED ] 3 tests. 2024-09-16T03:51:04.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-16T03:51:04.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-16T03:51:04.598 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: seed 39664 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (248 ms) 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (36 ms) 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (8 ms) 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-16T03:51:04.599 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (7 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T03:51:04.600 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (12 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (10 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (7 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (14 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (5 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T03:51:04.601 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (11 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (401 ms total) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1094 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (19 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-09-16T03:51:04.602 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T03:51:04.603 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (8 ms) 2024-09-16T03:51:05.659 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ally_mean_it": true}]: dispatch 2024-09-16T03:51:05.659 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:01.534262+0000 mon.a [INF] from='client.4774 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:05.659 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:03.626455+0000 mon.b [INF] from='client.? 172.21.15.70:0/3464022421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi070-40863-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:03.657292+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi070-40863-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:03.657779+0000 mon.a [INF] from='client.? 172.21.15.70:0/295175824' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi070-39867-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi070-39867-10"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466754+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi070-39720-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi070-39720-10"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466814+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi070-40183-1"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi070-39848-4", "tierpool": "test-rados-api-smithi070-39848-4-cache"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466902+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi070-39637-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466950+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi070-40921-1"}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.466989+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi070-39615-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.467035+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi070-40863-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.474051+0000 mon.c [INF] from='client.? 172.21.15.70:0/3879750091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.477877+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:05.660 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.483975+0000 mon.b [INF] from='client.? 172.21.15.70:0/657650792' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi070-39848-4", "overlaypool": "test-rados-api-smithi070-39848-4-cache"}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.484946+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi070-39848-4", "overlaypool": "test-rados-api-smithi070-39848-4-cache"}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.485419+0000 mon.a [INF] from='client.? 172.21.15.70:0/3907392717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.485587+0000 mon.b [INF] from='client.? 172.21.15.70:0/3464022421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi070-40863-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi070-40863-2"}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.486366+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi070-40863-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi070-40863-2"}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.553358+0000 mon.c [INF] from='client.? 172.21.15.70:0/2040604100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi070-40921-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:04.568958+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi070-40921-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.466956+0000 mon.a [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.472901+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-40329-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.472962+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi070-39848-4", "overlaypool": "test-rados-api-smithi070-39848-4-cache"}]': finished 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.473010+0000 mon.a [INF] from='client.? 172.21.15.70:0/3907392717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi070-39915-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:51:05.662 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.473058+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi070-40921-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:51:05.663 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.480610+0000 mon.b [INF] from='client.? 172.21.15.70:0/657650792' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi070-39848-4-cache", "mode": "writeback"}]: dispatch 2024-09-16T03:51:05.663 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.483375+0000 mon.c [INF] from='client.? 172.21.15.70:0/2040604100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi070-40921-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi070-40921-2"}]: dispatch 2024-09-16T03:51:05.663 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.504889+0000 mon.c [INF] from='client.? 172.21.15.70:0/2423522876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:06.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.507103+0000 mon.b [INF] from='client.? 172.21.15.70:0/2227015268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7"}]: d list: Running main() from gmock_main.cc 2024-09-16T03:51:06.512 INFO:tasks.workunit.client.0.smithi070.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [----------] Global test environment set-up. 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [----------] 3 tests from NeoradosList 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ OK ] NeoradosList.ListObjects (2776 ms) 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3052 ms) 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ OK ] NeoradosList.ListObjectsMany (2980 ms) 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [----------] 3 tests from NeoradosList (8808 ms total) 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [----------] Global test environment tear-down 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [==========] 3 tests from 1 test suite ran. (8809 ms total) 2024-09-16T03:51:06.513 INFO:tasks.workunit.client.0.smithi070.stdout: list: [ PASSED ] 3 tests. 2024-09-16T03:51:06.874 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:51:06.874 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T03:51:06.874 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T03:51:06.874 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5089 ms) 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ OK ] LibRadosService.RegisterLate (42 ms) 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: cluster: 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: id: 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 14 pool(s) do not have an application enabled 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: services: 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 88s) 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mgr: y(active, since 84s), standbys: x 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 86s) 2024-09-16T03:51:06.875 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: data: 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pools: 24 pools, 641 pgs 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: objects: 11 objects, 577 KiB 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: usage: 226 MiB used, 720 GiB / 720 GiB avail 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pgs: 84.399% pgs unknown 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 7.956% pgs not active 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 541 unknown 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 49 active+clean 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 42 creating+peering 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 9 creating+activating 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.876 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: io: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: client: 0 B/s rd, 3.7 KiB/s wr, 0 op/s rd, 7 op/s wr 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: cluster: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: id: 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 16 pool(s) do not have an application enabled 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: services: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 90s) 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mgr: y(active, since 86s), standbys: x 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: osd: 8 osds: 8 up (since 78s), 8 in (since 88s) 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.877 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: data: 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pools: 28 pools, 841 pgs 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: objects: 38 objects, 579 KiB 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: usage: 236 MiB used, 720 GiB / 720 GiB avail 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pgs: 45.422% pgs unknown 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 23.424% pgs not active 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 382 unknown 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 262 active+clean 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 190 creating+peering 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 7 creating+activating 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: io: 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: client: 0 B/s rd, 17 KiB/s wr, 169 op/s rd, 197 op/s wr 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.878 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: cluster: 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: id: 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 1 cache pools are missing hit_sets 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 18 pool(s) do not have an application enabled 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: services: 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 92s) 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: mgr: y(active, since 88s), standbys: x 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: osd: 8 osds: 8 up (since 80s), 8 in (since 90s) 2024-09-16T03:51:06.879 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T03:51:07.511 INFO:tasks.workunit.client.0.smithi070.stdout: ap api_io: Running main() from gmock_main.cc 2024-09-16T03:51:07.511 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T03:51:07.511 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T03:51:07.511 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T03:51:07.511 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (191 ms) 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: no timeout :/ 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: no timeout :/ 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: no timeout :/ 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: no timeout :/ 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: no timeout :/ 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (49 ms) 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (13 ms) 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T03:51:07.512 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (12 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (7 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (3 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T03:51:07.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.RmXattr (21 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIo.XattrIter (12 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [----------] 14 tests from LibRadosIo (330 ms total) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1104 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (12 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (19 ms) 2024-09-16T03:51:07.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (13 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (12 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (6 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (11 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (18 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1202 ms total) 2024-09-16T03:51:07.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: api_stat: Running main() from gmock_main.cc 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStat.Stat (168 ms) 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (13 ms) 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T03:51:07.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStat.StatNS (18 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (2 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (9 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] 5 tests from LibRadosStat (210 ms total) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1088 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (32 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-16T03:51:07.518 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (5 ms) 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1125 ms total) 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10116 ms total) 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T03:51:07.519 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: seed 40031 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (193 ms) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (72 ms) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (10 ms) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (19 ms) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (295 ms total) 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: 2024-09-16T03:51:07.520 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1084 ms) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (10 ms) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (34 ms) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1128 ms total) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10148 ms total) 2024-09-16T03:51:07.521 INFO:tasks.workunit.client.0.smithi070.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T03:51:07.522 INFO:tasks.workunit.client.0.smithi070.stdout: [----------] Global test environment tear-down 2024-09-16T03:51:07.522 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [==========] 24 tests from 2 test suites ran. (10286 ms total) 2024-09-16T03:51:07.522 INFO:tasks.workunit.client.0.smithi070.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (8 ms) 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (209 ms) 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T03:51:07.526 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3 ms) 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (5 ms) 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-09-16T03:51:07.527 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (15 ms) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (7 ms) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1398 ms total) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (10297 ms total) 2024-09-16T03:51:07.528 INFO:tasks.workunit.client.0.smithi070.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout:ispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.508681+0000 mon.b [INF] from='client.? 172.21.15.70:0/1433020878' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi070-40183-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.508685+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi070-39848-4-cache", "mode": "writeback"}]: dispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.509666+0000 mon.b [INF] from='client.4688 172.21.15.70:0/1029088860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.536712+0000 mon.a [INF] from='client.4810 172.21.15.70:0/1842986467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi070-40116-7"}]: dispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.537225+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi070-40921-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi070-40921-2"}]: dispatch 2024-09-16T03:51:09.668 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.537905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi070-40031-7"}]: dispatch 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.538171+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi070-40183-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.538424+0000 mon.a [INF] from='client.4688 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi070-39646-16"}]: dispatch 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.538735+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi070-39664-22"}]: dispatch 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.658202+0000 client.admin [INF] threexx 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: got: 2024-09-16T03:51:05.658436+0000 mon.a [INF] from='client.? 172.21.15.70:0/295175824' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9152 ms) 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (12245 ms total) 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (12245 ms total) 2024-09-16T03:51:09.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T03:51:10.352 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: Running main() from gmock_main.cc 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [----------] Global test environment set-up. 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5788 ms) 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6860 ms) 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12648 ms total) 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: 2024-09-16T03:51:10.353 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [----------] Global test environment tear-down 2024-09-16T03:51:10.354 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12648 ms total) 2024-09-16T03:51:10.354 INFO:tasks.workunit.client.0.smithi070.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-16T03:51:10.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T03:51:10.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:51:10.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T03:51:10.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T03:51:10.368 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: seed 39867 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (102 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (3 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1125 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1031 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (6 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (14 ms) 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T03:51:10.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (17 ms) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5 ms) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2303 ms total) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1101 ms) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4 ms) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1075 ms) 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T03:51:10.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1006 ms) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (8 ms) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (6 ms) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3208 ms total) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13056 ms total) 2024-09-16T03:51:10.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (68 ms) 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.LockShared (19 ms) 2024-09-16T03:51:10.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1131 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1024 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.Unlock (7 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (15 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (10 ms) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] 8 tests from LibRadosLock (2279 ms total) 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: 2024-09-16T03:51:10.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1097 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (4 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1057 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1009 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (5 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (7 ms) 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T03:51:10.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (9 ms) 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (5 ms) 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3193 ms total) 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13119 ms total) 2024-09-16T03:51:10.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1115 ms) 2024-09-16T03:51:10.858 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (11 ms) 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1126 ms total) 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (108 ms) 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3821 ms) 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (4 ms) 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T03:51:10.859 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (206 ms) 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196331008 notify_id 154618822657 notifier_gid 5032 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (7 ms) 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822658 notifier_gid 5032 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822659 notifier_gid 5032 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822658 notifier_gid 5032 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822659 notifier_gid 5032 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (207 ms) 2024-09-16T03:51:10.860 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822659 notifier_gid 5032 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (5 ms) 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822660 notifier_gid 5032 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T03:51:10.861 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: trying... 2024-09-16T03:51:11.390 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 154618822661 notif pool: Running main() from gmock_main.cc 2024-09-16T03:51:11.390 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-16T03:51:11.390 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [----------] Global test environment set-up. 2024-09-16T03:51:11.390 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-16T03:51:11.390 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolList (1829 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (1939 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2055 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3967 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1841 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2039 ms) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [----------] 6 tests from NeoRadosPools (13670 ms total) 2024-09-16T03:51:11.391 INFO:tasks.workunit.client.0.smithi070.stdout: pool: 2024-09-16T03:51:11.392 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [----------] Global test environment tear-down 2024-09-16T03:51:11.392 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [==========] 6 tests from 1 test suite ran. (13670 ms total) 2024-09-16T03:51:11.392 INFO:tasks.workunit.client.0.smithi070.stdout: pool: [ PASSED ] 6 tests. 2024-09-16T03:51:16.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-16T03:51:16.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T03:51:16.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T03:51:16.382 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (198 ms) 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94387178996624 err -107 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (12 ms) 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94387179008400 err -107 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (17 ms) 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182400 cookie 94387179026208 2024-09-16T03:51:16.383 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (22 ms) 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182400 cookie 94387179026208 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (13 ms) 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182401 cookie 94387179055232 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (14 ms) 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182400 cookie 94387179055232 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182400 cookie 94387179068240 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (17 ms) 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 107374182400 cookie 94387179068240 2024-09-16T03:51:16.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 111669149697 cookie 94387179068240 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3007 ms) 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94387179068240 err -107 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_cb from 4512 notify_id 141733920770 cookie 94387179068240 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94387179041248 err -107 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9312 ms total) 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: 2024-09-16T03:51:16.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1044 ms) 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1044 ms total) 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (18975 ms total) 2024-09-16T03:51:16.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T03:51:19.409 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: Running main() from gmock_main.cc 2024-09-16T03:51:19.409 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:51:19.409 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [----------] Global test environment set-up. 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6761 ms) 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6859 ms) 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8045 ms) 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [----------] 3 tests from NeoradosECList (21665 ms total) 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [----------] Global test environment tear-down 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (21665 ms total) 2024-09-16T03:51:19.410 INFO:tasks.workunit.client.0.smithi070.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T03:51:20.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3014 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3029 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2978 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2827 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3040 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2966 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5041 ms) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (22895 ms total) 2024-09-16T03:51:20.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: 2024-09-16T03:51:20.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T03:51:20.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (22895 ms total) 2024-09-16T03:51:20.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout:ier_gid 5032 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: timed out 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushing 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushed 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-09-16T03:51:26.463 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: trying... 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 167503724548 notifier_gid 5032 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: timed out 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushing 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushed 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3007 ms) 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: List watches 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify2 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 180388626437 notifier_gid 5032 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:51:26.464 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushing 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: done 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3012 ms) 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: List watches 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify2 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: handle_notify cookie 94685196345456 notify_id 193273528326 notifier_gid 5032 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: flushing 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: done 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16408 ms total) 2024-09-16T03:51:26.465 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: 2024-09-16T03:51:26.466 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T03:51:26.466 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28980 ms total) 2024-09-16T03:51:26.466 INFO:tasks.workunit.client.0.smithi070.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T03:51:26.938 INFO:tasks.workunit.client.0.smithi070.stdout:i_service: laundry: 1 daemon active (1 hosts) 2024-09-16T03:51:26.938 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:26.938 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: data: 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pools: 31 pools, 793 pgs 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: objects: 48 objects, 581 KiB 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: usage: 241 MiB used, 720 GiB / 720 GiB avail 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: pgs: 49.306% pgs unknown 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 4.918% pgs not active 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 391 unknown 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 363 active+clean 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 32 creating+peering 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 7 creating+activating 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: io: 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: client: 2.2 KiB/s rd, 18 KiB/s wr, 171 op/s rd, 198 op/s wr 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:26.939 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4260 ms) 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ OK ] LibRadosService.Status (20032 ms) 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [----------] 4 tests from LibRadosService (29423 ms total) 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29423 ms total) 2024-09-16T03:51:26.940 INFO:tasks.workunit.client.0.smithi070.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T03:51:28.100 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:51:28.100 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T03:51:28.100 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T03:51:28.100 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T03:51:28.100 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5116 ms) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (81 ms) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20059 ms) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: attempt 0 of 20 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5334 ms) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30590 ms total) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30591 ms total) 2024-09-16T03:51:28.101 INFO:tasks.workunit.client.0.smithi070.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T03:51:30.414 INFO:tasks.workunit.client.0.smithi070.stdout: misc: Running main() from gmock_main.cc 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [----------] Global test environment set-up. 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.Version (1827 ms) 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1943 ms) 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.LongName (3006 ms) 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3024 ms) 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-16T03:51:30.415 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2848 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3014 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.Exec (3001 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2982 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3049 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2951 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2031 ms) 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-16T03:51:30.416 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2985 ms) 2024-09-16T03:51:30.417 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [----------] 12 tests from NeoRadosMisc (32662 ms total) 2024-09-16T03:51:30.417 INFO:tasks.workunit.client.0.smithi070.stdout: misc: 2024-09-16T03:51:30.417 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [----------] Global test environment tear-down 2024-09-16T03:51:30.417 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [==========] 12 tests from 1 test suite ran. (32662 ms total) 2024-09-16T03:51:30.417 INFO:tasks.workunit.client.0.smithi070.stdout: misc: [ PASSED ] 12 tests. 2024-09-16T03:51:35.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T03:51:35.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3200 ms) 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (2952 ms) 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3038 ms) 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2844 ms) 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T03:51:35.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3026 ms) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5023 ms) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5007 ms) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4987 ms) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7948 ms) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [----------] 9 tests from LibRadosPools (38025 ms total) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (38027 ms total) 2024-09-16T03:51:35.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T03:51:38.391 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T03:51:38.391 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T03:51:38.391 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:50:57.353+0000 7f1d5728f780 -1 failed for service _ceph-mon._tcp 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:50:57.353+0000 7f1d5728f780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (60 ms) 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T03:51:38.392 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5014 ms) 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5074 ms total) 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: started 0x559359077850 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: started 0x559359cc6ee0 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: started 2 aios 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: waiting 0x559359077850 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: waiting 0x559359cc6ee0 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: done. 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5149 ms) 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5149 ms total) 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:51:38.393 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.Exec (78 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (14 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4894 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T03:51:38.394 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (23106 ms) 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28094 ms total) 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (41069 ms total) 2024-09-16T03:51:38.395 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T03:51:39.361 INFO:tasks.workunit.client.0.smithi070.stdout: io: Running main() from gmock_main.cc 2024-09-16T03:51:39.361 INFO:tasks.workunit.client.0.smithi070.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:51:39.361 INFO:tasks.workunit.client.0.smithi070.stdout: io: [----------] Global test environment set-up. 2024-09-16T03:51:39.361 INFO:tasks.workunit.client.0.smithi070.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-16T03:51:39.361 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.Limits (2835 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3055 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.ReadOp (2973 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.SparseRead (2827 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3019 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2984 ms) 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:51:39.362 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3036 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3038 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2970 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.Trunc (2988 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.Remove (2992 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3021 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.RmXattr (2984 ms) 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-16T03:51:39.363 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2980 ms) 2024-09-16T03:51:39.364 INFO:tasks.workunit.client.0.smithi070.stdout: io: [----------] 14 tests from NeoRadosIo (41703 ms total) 2024-09-16T03:51:39.364 INFO:tasks.workunit.client.0.smithi070.stdout: io: 2024-09-16T03:51:39.364 INFO:tasks.workunit.client.0.smithi070.stdout: io: [----------] Global test environment tear-down 2024-09-16T03:51:39.364 INFO:tasks.workunit.client.0.smithi070.stdout: io: [==========] 14 tests from 1 test suite ran. (41703 ms total) 2024-09-16T03:51:39.364 INFO:tasks.workunit.client.0.smithi070.stdout: io: [ PASSED ] 14 tests. 2024-09-16T03:51:39.373 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: Running main() from gmock_main.cc 2024-09-16T03:51:39.373 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:51:39.373 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [----------] Global test environment set-up. 2024-09-16T03:51:39.373 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-16T03:51:39.373 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2747 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3049 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2994 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2819 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3011 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2981 ms) 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-16T03:51:39.374 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3048 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2982 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (2969 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3054 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2974 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2982 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3023 ms) 2024-09-16T03:51:39.375 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2983 ms) 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41616 ms total) 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [----------] Global test environment tear-down 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41617 ms total) 2024-09-16T03:51:39.376 INFO:tasks.workunit.client.0.smithi070.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-16T03:51:53.363 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2008 ms) 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2063 ms) 2024-09-16T03:51:53.364 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1980 ms) 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1982 ms) 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8033 ms total) 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4035 ms) 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4002 ms) 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T03:51:53.365 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5016 ms) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13054 ms total) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2993 ms) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1999 ms) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1982 ms) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2008 ms) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8982 ms total) 2024-09-16T03:51:53.366 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4023 ms) 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3984 ms) 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8007 ms total) 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (56021 ms total) 2024-09-16T03:51:53.367 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T03:52:06.397 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: seed 40036 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1930 ms) 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2057 ms) 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2013 ms) 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T03:52:06.398 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1951 ms) 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3882 ms) 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11833 ms total) 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4054 ms) 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T03:52:06.399 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3984 ms) 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5983 ms) 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4013 ms) 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2036 ms) 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:06.400 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi070-40036-7 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17948 ms) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38018 ms total) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (10 ms) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5032 ms) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5042 ms total) 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:06.401 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T03:52:06.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T03:52:06.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3011 ms) 2024-09-16T03:52:06.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T03:52:08.354 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapsh api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T03:52:08.354 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: seed 39872 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (283 ms) 2024-09-16T03:52:08.355 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (15 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (8 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (14 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (11 ms) 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T03:52:08.356 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (7 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (8 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (9 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2492 ms) 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T03:52:08.357 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (362 ms) 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: done waiting 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60405 ms) 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4312 ms) 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T03:52:08.358 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T03:52:08.359 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T03:52:08.359 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-16T03:52:08.359 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T03:52:08.359 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T03:52:15.467 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=6 expected=6 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=8 expected=8 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=5 expected=5 2024-09-16T03:52:15.468 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=11 expected=11 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=15 expected=15 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=0 expected=0 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-09-16T03:52:15.469 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=14 expected=14 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: : entry=7 expected=7 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (177 ms) 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (53491 ms) 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-09-16T03:52:15.470 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (18032 ms) 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 7 tests from LibRadosList (71948 ms total) 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1022 ms) 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo1 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo2 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo3 2024-09-16T03:52:15.471 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo4 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo5 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo1 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset foo6,foo7 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo7 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: foo6 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo4 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo5 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns2:foo7 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns2:foo6 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: ns1:foo1 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo1 2024-09-16T03:52:15.472 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo2 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: :foo3 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (43 ms) 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 1 0 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 10 0 2024-09-16T03:52:15.473 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 13 0 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: api_aio: Running main() from gmock_main.cc 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2293 ms) 2024-09-16T03:52:21.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3074 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2928 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2837 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3046 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2958 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3057 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2983 ms) 2024-09-16T03:52:21.374 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3035 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2978 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2985 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2984 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3004 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2989 ms) 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T03:52:21.375 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.Flush (3007 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2975 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3008 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2997 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3005 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3017 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2984 ms) 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T03:52:21.376 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3009 ms) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3026 ms) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2966 ms) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3002 ms) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3013 ms) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] 26 tests from LibRadosAio (77162 ms total) 2024-09-16T03:52:21.377 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: 2024-09-16T03:52:21.378 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T03:52:21.378 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-16T03:52:21.379 INFO:tasks.workunit.client.0.smithi070.stdout: 7 0 2024-09-16T03:52:21.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 14 0 2024-09-16T03:52:21.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 0 0 2024-09-16T03:52:21.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 15 0 2024-09-16T03:52:21.379 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 11 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 5 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 8 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 6 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 3 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 4 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 12 0 2024-09-16T03:52:21.380 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 9 0 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 2 0 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (53 ms) 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 3 tests from LibRadosListEC (1118 ms total) 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3004 ms) 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] 1 test from LibRadosListNP (3004 ms total) 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [==========] 11 tests from 3 test suites ran. (84105 ms total) 2024-09-16T03:52:21.381 INFO:tasks.workunit.client.0.smithi070.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T03:52:31.384 INFO:tasks.workunit.client.0.smithi070.stdout:otsECPP.SnapRemovePP (2019 ms) 2024-09-16T03:52:31.384 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1954 ms) 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2031 ms) 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9015 ms total) 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T03:52:31.385 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4013 ms) 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3984 ms) 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3994 ms) 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11991 ms total) 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: 2024-09-16T03:52:31.386 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T03:52:31.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (94009 ms total) 2024-09-16T03:52:31.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T03:52:31.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T03:52:31.387 INFO:tasks.workunit.client.0.smithi070.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout:(67949 ms total) 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (84 ms) 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (84 ms total) 2024-09-16T03:52:34.402 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (60 ms) 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (64 ms total) 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (22 ms) 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (11 ms) 2024-09-16T03:52:34.403 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (33 ms total) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (33 ms) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (6 ms) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (39 ms total) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1017 ms) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1017 ms total) 2024-09-16T03:52:34.404 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: 2024-09-16T03:52:34.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T03:52:34.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (97069 ms total) 2024-09-16T03:52:34.405 INFO:tasks.workunit.client.0.smithi070.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2317 ms) 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (20845 ms) 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T03:52:38.369 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6025 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2966 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3016 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2984 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2977 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3025 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2974 ms) 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T03:52:38.370 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3011 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2997 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3004 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3017 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2984 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3025 ms) 2024-09-16T03:52:38.371 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2957 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3015 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3007 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2998 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2993 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3028 ms) 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T03:52:38.372 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2984 ms) 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3006 ms) 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2991 ms) 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3019 ms) 2024-09-16T03:52:38.373 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T03:54:06.415 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6991 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6998 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7027 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6991 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6984 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7016 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6997 ms) 2024-09-16T03:54:06.416 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7001 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7009 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7000 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6999 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6991 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7005 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7006 ms) 2024-09-16T03:54:06.417 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6997 ms) 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7020 ms) 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112032 ms total) 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (189195 ms total) 2024-09-16T03:54:06.418 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: seed 39848 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (184 ms) 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10717 ms) 2024-09-16T03:54:25.429 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (38 ms) 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10939 ms total) 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7015 ms) 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7984 ms) 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10012 ms) 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: my_snaps [3] 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: my_snaps [4,3] 2024-09-16T03:54:25.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: promoting some heads 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: done waiting 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46013 ms) 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10016 ms) 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7994 ms) 2024-09-16T03:54:25.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7988 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8023 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9981 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9018 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10041 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12976 ms) 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T03:54:25.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7980 ms) 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8013 ms) 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12992 ms) 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8006 ms) 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T03:54:25.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8031 ms) 2024-09-16T03:54:40.669 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: ok, hit_set contains 253:602f83fe:::foo:head 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8999 ms) 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 0 ls 1726458881,0 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 1 ls 1726458881,0 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 2 ls 1726458881,0 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 3 ls 1726458881,0 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 4 ls 1726458881,0 2024-09-16T03:54:40.670 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 5 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 6 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 7 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 8 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 9 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 10 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 11 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 12 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 13 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 14 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 15 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 16 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 17 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 18 ls 1726458881,0 2024-09-16T03:54:40.671 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 19 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 20 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 21 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 22 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 23 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 24 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 25 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 26 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 27 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 28 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 29 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 30 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg 31 ls 1726458881,0 2024-09-16T03:54:40.672 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6cac518f:::0:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:02547ec2:::1:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f905c69b:::2:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:cfc208b3:::3:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d83876eb:::4:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b29083e3:::5:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c4fdafeb:::6:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:5c6b0b28:::7:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:bd63b0f1:::8:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e960b815:::9:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:52ea6a34:::10:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:89d3ae78:::11:head 2024-09-16T03:54:40.673 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:de5d7c5f:::12:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:566253c9:::13:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:62a1935d:::14:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:863748b0:::15:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3958e169:::16:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4d4dabf9:::17:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:8391935d:::18:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:28883081:::19:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:69259c59:::20:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4bdb80b7:::21:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a11c5d71:::22:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:271af37b:::23:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:95b121be:::24:head 2024-09-16T03:54:40.674 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:58d1031b:::25:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:0a050783:::26:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c709704c:::27:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:cbe56eaf:::28:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:86b4b162:::29:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:70d89383:::30:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:dd450c7c:::31:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6d5729b1:::32:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c388f3fb:::33:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:56cfea31:::34:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9dbc1bf7:::35:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:40b74ccd:::36:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4d5aaf42:::37:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:920f362c:::38:head 2024-09-16T03:54:40.675 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6cc53222:::39:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9cad833f:::40:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:1ea84d41:::41:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c4480ef6:::42:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a694361e:::43:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d1bd33e9:::44:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ddc2cd5d:::45:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:2b782207:::46:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7b187fca:::47:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:90ecdf6f:::48:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a5ed95fe:::49:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ea0eaa55:::50:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f33ef17b:::51:head 2024-09-16T03:54:40.676 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a0d1b2f6:::52:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:60c5229e:::53:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:edcbc575:::54:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:102cf253:::55:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:efb7fb0b:::56:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:50d0a326:::57:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d4dc5daf:::58:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3a130462:::59:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ec87ed71:::60:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d5bc9454:::61:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3ddfe313:::62:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7c2816b9:::63:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:47e00e4d:::64:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c6410c18:::65:head 2024-09-16T03:54:40.677 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b48ed237:::66:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:cd63ad31:::67:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b179e92b:::68:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:0d9f741a:::69:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6d3352ae:::70:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c6d5c19e:::71:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:bc4729c3:::72:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:77e930b9:::73:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:0abeecfd:::74:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b7c37e15:::75:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b6378398:::76:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:02bd68de:::77:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:cc795d2d:::78:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:630d4fea:::79:head 2024-09-16T03:54:40.678 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e0d29ef5:::80:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:fd6f13d2:::81:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:606461d5:::82:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:eadbdc43:::83:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:8761d0bb:::84:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9ef0186f:::85:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e0d41294:::86:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:961de695:::87:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:1423148f:::88:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:633a8fa2:::89:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a8653809:::90:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3dac8b33:::91:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:35aad435:::92:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f6dcc343:::93:head 2024-09-16T03:54:40.679 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:dbbdad87:::94:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:1cb48ce0:::95:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:03cd461c:::96:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:17a4ea99:::97:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9993c9a7:::98:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6394211c:::99:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:94c7ae57:::100:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6fdee5bb:::101:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9a477fd1:::102:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:eb850916:::103:head 2024-09-16T03:54:40.680 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:affc56b9:::104:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b42dc814:::105:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f319f8f0:::106:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9a40b9de:::107:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:8b524f28:::108:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e3de589f:::109:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:90f90a5b:::110:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a7b4f1d7:::111:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:af51766e:::112:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b6f90bd1:::113:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e0261208:::114:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c9569ef7:::115:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:61bebe50:::116:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:fe93412b:::117:head 2024-09-16T03:54:40.681 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d3d38bee:::118:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3100ba0c:::119:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d0560ada:::120:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f0ea8b35:::121:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:766f231a:::122:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a07a2582:::123:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:bd7c6b3a:::124:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:fb2ddaff:::125:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4408e1fe:::126:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ee1df7a7:::127:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c3002909:::128:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4f48ffa9:::129:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:edf38733:::130:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c08425c0:::131:head 2024-09-16T03:54:40.682 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:5f902d98:::132:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:41ea2c93:::133:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:813cee13:::134:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:0131818d:::135:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:26ba5a85:::136:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:381b8a5a:::137:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:28797e47:::138:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:bfca7f22:::139:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:36807075:::140:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:80b03975:::141:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:5c15709b:::142:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f39ea15e:::143:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ea992956:::144:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:48887b1c:::145:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:9f24a9dd:::146:head 2024-09-16T03:54:40.683 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:987f100b:::147:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d2dd3581:::148:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7fed1808:::149:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c80b70e9:::150:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:85ed90f9:::151:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:36428b24:::152:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d044c34a:::153:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7c18bf58:::154:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d1c21232:::155:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a7a3c575:::156:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:87da0633:::157:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d5ac3822:::158:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3f20522d:::159:head 2024-09-16T03:54:40.684 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6ca26563:::160:head 2024-09-16T03:54:40.685 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:532ce135:::161:head 2024-09-16T03:54:40.685 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:c78863e6:::162:head 2024-09-16T03:55:04.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7462ddf6:::.RoundTripAppendPP (2974 ms) 2024-09-16T03:55:04.430 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3005 ms) 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3029 ms) 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2985 ms) 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: setting pool EIO 2024-09-16T03:55:04.431 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: max_success 98, min_failed 99 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4006 ms) 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2988 ms) 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (117162 ms total) 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3004 ms) 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9008 ms) 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T03:55:04.432 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14998 ms) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3001 ms) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30012 ms total) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3040 ms) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3040 ms total) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13955 ms) 2024-09-16T03:55:04.433 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7001 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7009 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7006 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3004 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6995 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6999 ms) 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T03:55:04.434 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7032 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6980 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6987 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7029 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7003 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6976 ms) 2024-09-16T03:55:04.435 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-09-16T03:55:35.441 INFO:tasks.workunit.client.0.smithi070.stdout: 163:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:5d165639:::164:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f43765fc:::165:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b4c720e9:::166:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e694b040:::167:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:afa38db2:::168:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:77ba9f53:::169:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:87495034:::170:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7c96bf0e:::171:head 2024-09-16T03:55:35.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:dbe346cc:::172:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:e943ec24:::173:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f97a9c0c:::174:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:6f26e74d:::175:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4f95e106:::176:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:0e6f2f8f:::177:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:05db05f1:::178:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:38a78d66:::179:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:d095610b:::180:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:a1a9d709:::181:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:1e5d39db:::182:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:f7df4fb9:::183:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:03a7f161:::184:head 2024-09-16T03:55:35.443 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ba70721e:::185:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:28e5662d:::186:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:973d52de:::187:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:4303eb1c:::188:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:b990b48e:::189:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:29b8165b:::190:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:3547f197:::191:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:7e260936:::192:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:1abec7b1:::193:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:10fdda93:::194:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:15817eea:::195:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:770bab57:::196:head 2024-09-16T03:55:35.444 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:ed9e13e7:::197:head 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:71471a8f:::198:head 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: checking for 255:10fb1d02:::199:head 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8005 ms) 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: first is 1726458890 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,0 2024-09-16T03:55:35.445 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,1726458898,1726458899,0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,1726458898,1726458899,0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458890,1726458892,1726458893,1726458895,1726458896,1726458898,1726458899,0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726458893,1726458895,1726458896,1726458898,1726458899,1726458901,1726458902,0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: first now 1726458893, trimmed 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20992 ms) 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: foo0 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14002 ms) 2024-09-16T03:55:35.446 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3019 ms) 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6988 ms) 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6993 ms) 2024-09-16T03:55:46.441 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7026 ms) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6997 ms) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6984 ms) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7020 ms) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139004 ms total) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (289218 ms total) 2024-09-16T03:55:46.442 INFO:tasks.workunit.client.0.smithi070.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T03:58:29.516 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-16T03:58:29.513+0000 7f41442e4640 -1 snap_mapper.add_oid found existing snaps mapped on 284:ff4f7f38:test-rados-api-smithi070-39848-80::foo:2, removing 2024-09-16T03:58:29.517 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:58:29.513+0000 7fe0190f0640 -1 snap_mapper.add_oid found existing snaps mapped on 284:ff4f7f38:test-rados-api-smithi070-39848-80::foo:2, removing 2024-09-16T03:59:00.512 INFO:tasks.workunit.client.0.smithi070.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18015 ms) 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23026 ms) 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3027 ms) 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2935 ms) 2024-09-16T03:59:00.513 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3012 ms) 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3022 ms) 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4017 ms) 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38107 ms) 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16863 ms) 2024-09-16T03:59:00.514 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3992 ms) 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3004 ms) 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25028 ms) 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15046 ms) 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T03:59:00.515 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5036 ms) 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3909 ms) 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: done waiting 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25061 ms) 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9966 ms) 2024-09-16T03:59:00.516 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9002 ms) 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8986 ms) 2024-09-16T03:59:00.517 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T03:59:57.529 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-09-16T03:59:57.525+0000 7fe0208ff640 -1 snap_mapper.add_oid found existing snaps mapped on 98:13d795e8:test-rados-api-smithi070-39848-97::foo:21, removing 2024-09-16T03:59:57.530 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-09-16T03:59:57.525+0000 7f8310831640 -1 snap_mapper.add_oid found existing snaps mapped on 98:13d795e8:test-rados-api-smithi070-39848-97::foo:21, removing 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: Running main() from gmock_main.cc 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] Global test environment set-up. 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (502747 ms) 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4984 ms) 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4021 ms) 2024-09-16T04:00:29.535 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4983 ms) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7014 ms) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (523750 ms total) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4991 ms) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6045 ms) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7975 ms) 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-16T04:00:29.536 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5984 ms) 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4013 ms) 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi070-40997-11. 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: Waiting for snaps to purge. 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19028 ms) 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48036 ms total) 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [----------] Global test environment tear-down 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (571786 ms total) 2024-09-16T04:00:29.537 INFO:tasks.workunit.client.0.smithi070.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout:api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9011 ms) 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5977 ms) 2024-09-16T04:02:31.567 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5015 ms) 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25994 ms) 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14020 ms) 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12001 ms) 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T04:02:31.568 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 00000012 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12018 ms) 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5998 ms) 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (555170 ms total) 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (992 ms) 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T04:02:31.569 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10975 ms) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18068 ms) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30038 ms total) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6977 ms) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8002 ms) 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: waiting for scrub... 2024-09-16T04:02:31.570 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: done waiting 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24073 ms) 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9955 ms) 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7986 ms) 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7998 ms) 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10003 ms) 2024-09-16T04:02:31.571 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T04:05:02.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8044 ms) 2024-09-16T04:05:02.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T04:05:02.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13000 ms) 2024-09-16T04:05:02.593 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8053 ms) 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12919 ms) 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8000 ms) 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8038 ms) 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: ok, hit_set contains 326:602f83fe:::foo:head 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8989 ms) 2024-09-16T04:05:02.594 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: first is 1726459418 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,1726459426,1726459427,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,1726459426,1726459427,0 2024-09-16T04:05:02.595 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459418,1726459420,1726459421,1726459423,1726459424,1726459426,1726459427,0 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: got ls 1726459421,1726459423,1726459424,1726459426,1726459427,1726459429,1726459430,0 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: first now 1726459421, trimmed 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21067 ms) 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: foo0 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13975 ms) 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18068 ms) 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21936 ms) 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T04:05:02.596 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5011 ms) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3016 ms) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3015 ms) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2912 ms) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231039 ms total) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (845268 ms total) 2024-09-16T04:05:02.597 INFO:tasks.workunit.client.0.smithi070.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T04:05:02.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39586 2024-09-16T04:05:02.600 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39586 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39819 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39819 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40362 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40362 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40124 2024-09-16T04:05:02.601 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40124 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39912 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39912 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39677 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39677 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40013 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40013 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40467 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40467 2024-09-16T04:05:02.602 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40513 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40513 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39706 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39706 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40394 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40394 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40239 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40239 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39561 2024-09-16T04:05:02.603 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39561 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39627 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39627 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40315 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40315 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39552 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39552 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39572 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39572 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40697 2024-09-16T04:05:02.604 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40697 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39847 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39847 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40798 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40798 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39943 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39943 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40450 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40450 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:02.605 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40891 2024-09-16T04:05:02.606 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40891 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-16T04:21:05.824 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: handle_notify cookie 94216677685840 notify_id 3152505995270 notifier_gid 4550 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1801026 ms) 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Trying... 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: handle_notify cookie 94216690579824 notify_id 3165390897153 notifier_gid 20915 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Waiting for 3.000000000s 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Timed out. 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Flushing... 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: Flushed... 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7047 ms) 2024-09-16T04:21:05.825 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1808074 ms total) 2024-09-16T04:21:05.826 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: 2024-09-16T04:21:05.826 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-16T04:21:05.826 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1808074 ms total) 2024-09-16T04:21:05.826 INFO:tasks.workunit.client.0.smithi070.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39870 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39870 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40327 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40327 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40624 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40624 2024-09-16T04:21:05.827 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40424 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40424 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40768 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40768 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40213 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40213 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39771 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39771 2024-09-16T04:21:05.828 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40829 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40829 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40145 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40145 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39545 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39545 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39605 2024-09-16T04:21:05.829 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39605 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40072 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40072 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=39650 2024-09-16T04:21:05.830 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 39650 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40157 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40157 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40181 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40181 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40570 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40570 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:21:05.831 INFO:tasks.workunit.client.0.smithi070.stderr:+ pid=40925 2024-09-16T04:21:05.832 INFO:tasks.workunit.client.0.smithi070.stderr:+ wait 40925 2024-09-16T04:21:17.887 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: Running main() from gmock_main.cc 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [----------] Global test environment set-up. 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (1801005 ms) 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3016 ms) 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3028 ms) 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-16T04:21:17.888 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3007 ms) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3024 ms) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3017 ms) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (4021 ms) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (1820118 ms total) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [----------] Global test environment tear-down 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (1820118 ms total) 2024-09-16T04:21:17.889 INFO:tasks.workunit.client.0.smithi070.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-16T04:21:17.890 INFO:tasks.workunit.client.0.smithi070.stderr:+ exit 0 2024-09-16T04:21:17.890 INFO:tasks.workunit.client.0.smithi070.stderr:+ cleanup 2024-09-16T04:21:17.890 INFO:tasks.workunit.client.0.smithi070.stderr:+ pkill -P 39539 2024-09-16T04:21:17.990 INFO:tasks.workunit.client.0.smithi070.stderr:+ true 2024-09-16T04:21:17.991 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:21:17.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:21:18.024 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-16T04:21:18.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T04:21:18.358 DEBUG:teuthology.parallel:result is None 2024-09-16T04:21:18.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:21:18.383 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:21:18.384 DEBUG:teuthology.orchestra.run.smithi070:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T04:21:18.447 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:21:18.447 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:21:18.457 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:21:18.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:21:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.583+0000 7f1a35faf640 10 monclient: get_monmap_and_config 2024-09-16T04:21:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.583+0000 7f1a35faf640 10 monclient: build_initial_monmap 2024-09-16T04:21:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.583+0000 7f1a35faf640 10 monclient: monmap: 2024-09-16T04:21:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:18.583822+0000 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:18.583822+0000 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:18.585 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:18.586 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:18.586 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 Processor -- start 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 -- start start 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient: init 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-b con 0x7f1a300acb60 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a301418b0 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f1a3013e3a0 0x7f1a3014b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-c con 0x7f1a3013e3a0 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a300aeaf0 con 0x7f1a301418b0 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f1a300aedd0 con 0x7f1a3013e3a0 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f1a300af0b0 con 0x7f1a300acb60 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a35faf640 10 monclient(hunting): authenticate will time out at 2637.245743s 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54692/0 (socket says 172.21.15.70:54692) 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/90067427 learned_addr learned my addr 172.21.15.70:0/90067427 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a300acb60 auth_method 0 2024-09-16T04:21:18.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.584+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2ffff640 1 --2- 172.21.15.70:0/90067427 >> v2:172.21.15.163:3300/0 conn(0x7f1a3013e3a0 0x7f1a3014b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request con 0x7f1a3013e3a0 auth_method 0 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_done global_id 20945 payload 274 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/90067427 >> v2:172.21.15.163:3300/0 conn(0x7f1a3013e3a0 msgr2=0x7f1a3014b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 --2- 172.21.15.70:0/90067427 >> v2:172.21.15.163:3300/0 conn(0x7f1a3013e3a0 0x7f1a3014b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/90067427 >> v1:172.21.15.163:6789/0 conn(0x7f1a301418b0 legacy=0x7f1a3013dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 monclient: found mon.noname-b 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/90067427 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a30140e50 con 0x7f1a300acb60 2024-09-16T04:21:18.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2f7fe640 1 --2- 172.21.15.70:0/90067427 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009e50 tx=0x7f1a2002a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47b1fc94cdb45090 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 1 -- 172.21.15.70:0/90067427 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20036070 con 0x7f1a300acb60 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: dump: 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.601 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a35faf640 5 monclient: authenticate success, global_id 20945 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 1 -- 172.21.15.70:0/90067427 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a20031600 con 0x7f1a300acb60 2024-09-16T04:21:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a35faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a35faf640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a35faf640 4 set_mon_vals no callback set 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 1 -- 172.21.15.70:0/90067427 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a200319d0 con 0x7f1a300acb60 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.585+0000 7f1a2e7fc640 10 monclient: dump: 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.603 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 10 monclient: shutdown 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 -- 172.21.15.70:0/90067427 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 msgr2=0x7f1a30141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 --2- 172.21.15.70:0/90067427 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009e50 tx=0x7f1a2002a6c0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 -- 172.21.15.70:0/90067427 shutdown_connections 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 --2- 172.21.15.70:0/90067427 >> v2:172.21.15.163:3300/0 conn(0x7f1a3013e3a0 0x7f1a3014b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 --2- 172.21.15.70:0/90067427 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a30141370 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 -- 172.21.15.70:0/90067427 >> 172.21.15.70:0/90067427 conn(0x7f1a300b1830 msgr2=0x7f1a300b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 -- 172.21.15.70:0/90067427 shutdown_connections 2024-09-16T04:21:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.586+0000 7f1a35faf640 1 -- 172.21.15.70:0/90067427 wait complete. 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 10 monclient: build_initial_monmap 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 10 monclient: monmap: 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:18.605 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 librados: starting msgr at 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 librados: starting objecter 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 Processor -- start 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 -- start start 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 librados: setting wanted keys 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 1 librados: calling monclient init 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.587+0000 7f1a35faf640 10 monclient: init 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:18.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-c con 0x7f1a300acb60 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3013e3a0 0x7f1a301d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3013e3a0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a301418b0 0x7f1a301388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): picked mon.noname-a con 0x7f1a301418b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300acf40 con 0x7f1a301418b0 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300af7e0 con 0x7f1a3013e3a0 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f1a300ad0c0 con 0x7f1a300acb60 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient(hunting): authenticate will time out at 2637.249241s 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54694/0 (socket says 172.21.15.70:54694) 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/1288009237 learned_addr learned my addr 172.21.15.70:0/1288009237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request con 0x7f1a300acb60 auth_method 0 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a301418b0 0x7f1a301388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request con 0x7f1a301418b0 auth_method 0 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3013e3a0 0x7f1a301d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): get_auth_request con 0x7f1a3013e3a0 auth_method 0 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient(hunting): handle_auth_done global_id 20948 payload 824 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/1288009237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3013e3a0 msgr2=0x7f1a301d98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3013e3a0 0x7f1a301d98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a301418b0 msgr2=0x7f1a301388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a301418b0 0x7f1a301388f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 -- 172.21.15.70:0/1288009237 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a301cbed0 con 0x7f1a300acb60 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a2f7fe640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009e50 tx=0x7f1a20004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 client.?.objecter ms_handle_connect 0x7f1a300acb60 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20036050 con 0x7f1a300acb60 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 monclient: dump: 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.611 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 5 monclient: authenticate success, global_id 20948 2024-09-16T04:21:18.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient: _renew_subs 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a20042070 con 0x7f1a300acb60 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.588+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a301cce20 con 0x7f1a300acb60 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a27fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20009510 con 0x7f1a300acb60 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a27fff640 10 monclient: dump: 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a2dffb640 4 set_mon_vals no callback set 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a35faf640 10 client.20948.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a35faf640 10 monclient: _renew_subs 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a35faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a30144620 con 0x7f1a300acb60 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.589+0000 7f1a35faf640 1 librados: init done 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a35faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a35faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a35faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a30141370 con 0x7f1a300acb60 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f1a20007c70 con 0x7f1a300acb60 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.590+0000 7f1a27fff640 1 --2- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 0x7f1a100700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a2effd640 1 --2- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 0x7f1a100700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 694..754) ==== 5257+0+0 (secure 0 0 0) 0x7f1a200abfb0 con 0x7f1a300acb60 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 10 client.20948.objecter ms_dispatch 0x7f1a30000b90 osd_map(754..754 src has 694..754) 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 3 client.20948.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 3 client.20948.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a2effd640 10 monclient: get_auth_request con 0x7f1a1006dc00 auth_method 0 2024-09-16T04:21:18.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 20 client.20948.objecter dump_active .. 0 homeless 2024-09-16T04:21:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a2effd640 1 --2- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 0x7f1a100700c0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f1a301da970 tx=0x7f1a18009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.591+0000 7f1a27fff640 10 client.20948.objecter ms_handle_connect 0x7f1a1006dc00 2024-09-16T04:21:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.595+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1a2007a2a0 con 0x7f1a300acb60 2024-09-16T04:21:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.595+0000 7f1a27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.595+0000 7f1a27fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:18.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.763+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a300b1200 con 0x7f1a1006dc00 2024-09-16T04:21:18.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.767+0000 7f1a27fff640 1 -- 172.21.15.70:0/1288009237 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116304 (secure 0 0 0) 0x7f1a300b1200 con 0x7f1a1006dc00 2024-09-16T04:21:18.770 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:18.771 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 10 librados: watch_flush enter 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 10 librados: watch_flush exit 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 20 client.20948.objecter shutdown clearing up homeless session... 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 10 client.20948.objecter successfully canceled tick 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 msgr2=0x7f1a100700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 0x7f1a100700c0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f1a301da970 tx=0x7f1a18009060 comp rx=0 tx=0).stop 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 10 monclient: shutdown 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 msgr2=0x7f1a301cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009e50 tx=0x7f1a20004a40 comp rx=0 tx=0).stop 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 shutdown_connections 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> v2:172.21.15.70:6804/198622594 conn(0x7f1a1006dc00 0x7f1a100700c0 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a301418b0 0x7f1a301388f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a3013e3a0 0x7f1a301d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 --2- 172.21.15.70:0/1288009237 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1a300acb60 0x7f1a301cc8e0 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 >> 172.21.15.70:0/1288009237 conn(0x7f1a300b1830 msgr2=0x7f1a300b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 shutdown_connections 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.769+0000 7f1a35faf640 1 -- 172.21.15.70:0/1288009237 wait complete. 2024-09-16T04:21:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.770+0000 7f1a35faf640 1 librados: shutdown 2024-09-16T04:21:18.782 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1693,"stamp":"2024-09-16T04:21:17.878077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":588668,"kb_used_data":35556,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754386052,"statfs":{"total":773094113280,"available":772491317248,"internally_reserved":0,"allocated":36409344,"data_stored":17363421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":563793458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020319"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878232+0000","last_change":"2024-09-16T03:50:58.527921+0000","last_active":"2024-09-16T04:21:16.878232+0000","last_peered":"2024-09-16T04:21:16.878232+0000","last_clean":"2024-09-16T04:21:16.878232+0000","last_became_active":"2024-09-16T03:50:58.527591+0000","last_became_peered":"2024-09-16T03:50:58.527591+0000","last_unstale":"2024-09-16T04:21:16.878232+0000","last_undegraded":"2024-09-16T04:21:16.878232+0000","last_fullsized":"2024-09-16T04:21:16.878232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.155864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872367+0000","last_change":"2024-09-16T03:50:58.566215+0000","last_active":"2024-09-16T04:21:15.872367+0000","last_peered":"2024-09-16T04:21:15.872367+0000","last_clean":"2024-09-16T04:21:15.872367+0000","last_became_active":"2024-09-16T03:50:58.565169+0000","last_became_peered":"2024-09-16T03:50:58.565169+0000","last_unstale":"2024-09-16T04:21:15.872367+0000","last_undegraded":"2024-09-16T04:21:15.872367+0000","last_fullsized":"2024-09-16T04:21:15.872367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:31.209733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872690+0000","last_change":"2024-09-16T03:50:58.521189+0000","last_active":"2024-09-16T04:21:15.872690+0000","last_peered":"2024-09-16T04:21:15.872690+0000","last_clean":"2024-09-16T04:21:15.872690+0000","last_became_active":"2024-09-16T03:50:58.520920+0000","last_became_peered":"2024-09-16T03:50:58.520920+0000","last_unstale":"2024-09-16T04:21:15.872690+0000","last_undegraded":"2024-09-16T04:21:15.872690+0000","last_fullsized":"2024-09-16T04:21:15.872690+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:15.187842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878466+0000","last_change":"2024-09-16T03:50:58.529897+0000","last_active":"2024-09-16T04:21:16.878466+0000","last_peered":"2024-09-16T04:21:16.878466+0000","last_clean":"2024-09-16T04:21:16.878466+0000","last_became_active":"2024-09-16T03:50:58.529616+0000","last_became_peered":"2024-09-16T03:50:58.529616+0000","last_unstale":"2024-09-16T04:21:16.878466+0000","last_undegraded":"2024-09-16T04:21:16.878466+0000","last_fullsized":"2024-09-16T04:21:16.878466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:40.902750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870995+0000","last_change":"2024-09-16T03:50:58.547657+0000","last_active":"2024-09-16T04:21:16.870995+0000","last_peered":"2024-09-16T04:21:16.870995+0000","last_clean":"2024-09-16T04:21:16.870995+0000","last_became_active":"2024-09-16T03:50:58.547418+0000","last_became_peered":"2024-09-16T03:50:58.547418+0000","last_unstale":"2024-09-16T04:21:16.870995+0000","last_undegraded":"2024-09-16T04:21:16.870995+0000","last_fullsized":"2024-09-16T04:21:16.870995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:45.151909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871082+0000","last_change":"2024-09-16T03:50:58.563918+0000","last_active":"2024-09-16T04:21:16.871082+0000","last_peered":"2024-09-16T04:21:16.871082+0000","last_clean":"2024-09-16T04:21:16.871082+0000","last_became_active":"2024-09-16T03:50:58.563681+0000","last_became_peered":"2024-09-16T03:50:58.563681+0000","last_unstale":"2024-09-16T04:21:16.871082+0000","last_undegraded":"2024-09-16T04:21:16.871082+0000","last_fullsized":"2024-09-16T04:21:16.871082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:01.363054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871296+0000","last_change":"2024-09-16T03:50:58.563162+0000","last_active":"2024-09-16T04:21:16.871296+0000","last_peered":"2024-09-16T04:21:16.871296+0000","last_clean":"2024-09-16T04:21:16.871296+0000","last_became_active":"2024-09-16T03:50:58.562917+0000","last_became_peered":"2024-09-16T03:50:58.562917+0000","last_unstale":"2024-09-16T04:21:16.871296+0000","last_undegraded":"2024-09-16T04:21:16.871296+0000","last_fullsized":"2024-09-16T04:21:16.871296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:20.514696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871351+0000","last_change":"2024-09-16T03:50:58.552412+0000","last_active":"2024-09-16T04:21:16.871351+0000","last_peered":"2024-09-16T04:21:16.871351+0000","last_clean":"2024-09-16T04:21:16.871351+0000","last_became_active":"2024-09-16T03:50:58.552137+0000","last_became_peered":"2024-09-16T03:50:58.552137+0000","last_unstale":"2024-09-16T04:21:16.871351+0000","last_undegraded":"2024-09-16T04:21:16.871351+0000","last_fullsized":"2024-09-16T04:21:16.871351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:13.498957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871506+0000","last_change":"2024-09-16T03:50:58.546028+0000","last_active":"2024-09-16T04:21:16.871506+0000","last_peered":"2024-09-16T04:21:16.871506+0000","last_clean":"2024-09-16T04:21:16.871506+0000","last_became_active":"2024-09-16T03:50:58.545776+0000","last_became_peered":"2024-09-16T03:50:58.545776+0000","last_unstale":"2024-09-16T04:21:16.871506+0000","last_undegraded":"2024-09-16T04:21:16.871506+0000","last_fullsized":"2024-09-16T04:21:16.871506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:03.340804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1442,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879550+0000","last_change":"2024-09-16T03:50:58.540624+0000","last_active":"2024-09-16T04:21:15.879550+0000","last_peered":"2024-09-16T04:21:15.879550+0000","last_clean":"2024-09-16T04:21:15.879550+0000","last_became_active":"2024-09-16T03:50:58.540379+0000","last_became_peered":"2024-09-16T03:50:58.540379+0000","last_unstale":"2024-09-16T04:21:15.879550+0000","last_undegraded":"2024-09-16T04:21:15.879550+0000","last_fullsized":"2024-09-16T04:21:15.879550+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:12.307201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879223+0000","last_change":"2024-09-16T03:50:58.480959+0000","last_active":"2024-09-16T04:21:16.879223+0000","last_peered":"2024-09-16T04:21:16.879223+0000","last_clean":"2024-09-16T04:21:16.879223+0000","last_became_active":"2024-09-16T03:50:58.480684+0000","last_became_peered":"2024-09-16T03:50:58.480684+0000","last_unstale":"2024-09-16T04:21:16.879223+0000","last_undegraded":"2024-09-16T04:21:16.879223+0000","last_fullsized":"2024-09-16T04:21:16.879223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:38.866093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1460,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872748+0000","last_change":"2024-09-16T03:50:58.508324+0000","last_active":"2024-09-16T04:21:16.872748+0000","last_peered":"2024-09-16T04:21:16.872748+0000","last_clean":"2024-09-16T04:21:16.872748+0000","last_became_active":"2024-09-16T03:50:58.507923+0000","last_became_peered":"2024-09-16T03:50:58.507923+0000","last_unstale":"2024-09-16T04:21:16.872748+0000","last_undegraded":"2024-09-16T04:21:16.872748+0000","last_fullsized":"2024-09-16T04:21:16.872748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:50.490187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871014+0000","last_change":"2024-09-16T03:50:58.539556+0000","last_active":"2024-09-16T04:21:16.871014+0000","last_peered":"2024-09-16T04:21:16.871014+0000","last_clean":"2024-09-16T04:21:16.871014+0000","last_became_active":"2024-09-16T03:50:58.539098+0000","last_became_peered":"2024-09-16T03:50:58.539098+0000","last_unstale":"2024-09-16T04:21:16.871014+0000","last_undegraded":"2024-09-16T04:21:16.871014+0000","last_fullsized":"2024-09-16T04:21:16.871014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:34.294689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":1616,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871130+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T04:21:16.871130+0000","last_peered":"2024-09-16T04:21:16.871130+0000","last_clean":"2024-09-16T04:21:16.871130+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T04:21:16.871130+0000","last_undegraded":"2024-09-16T04:21:16.871130+0000","last_fullsized":"2024-09-16T04:21:16.871130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":1457,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878405+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T04:21:16.878405+0000","last_peered":"2024-09-16T04:21:16.878405+0000","last_clean":"2024-09-16T04:21:16.878405+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T04:21:16.878405+0000","last_undegraded":"2024-09-16T04:21:16.878405+0000","last_fullsized":"2024-09-16T04:21:16.878405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879231+0000","last_change":"2024-09-16T03:50:58.527360+0000","last_active":"2024-09-16T04:21:16.879231+0000","last_peered":"2024-09-16T04:21:16.879231+0000","last_clean":"2024-09-16T04:21:16.879231+0000","last_became_active":"2024-09-16T03:50:58.527023+0000","last_became_peered":"2024-09-16T03:50:58.527023+0000","last_unstale":"2024-09-16T04:21:16.879231+0000","last_undegraded":"2024-09-16T04:21:16.879231+0000","last_fullsized":"2024-09-16T04:21:16.879231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:42.928632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871059+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T04:21:16.871059+0000","last_peered":"2024-09-16T04:21:16.871059+0000","last_clean":"2024-09-16T04:21:16.871059+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T04:21:16.871059+0000","last_undegraded":"2024-09-16T04:21:16.871059+0000","last_fullsized":"2024-09-16T04:21:16.871059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871813+0000","last_change":"2024-09-16T03:50:58.528877+0000","last_active":"2024-09-16T04:21:16.871813+0000","last_peered":"2024-09-16T04:21:16.871813+0000","last_clean":"2024-09-16T04:21:16.871813+0000","last_became_active":"2024-09-16T03:50:58.528663+0000","last_became_peered":"2024-09-16T03:50:58.528663+0000","last_unstale":"2024-09-16T04:21:16.871813+0000","last_undegraded":"2024-09-16T04:21:16.871813+0000","last_fullsized":"2024-09-16T04:21:16.871813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.620543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1463,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871100+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T04:21:16.871100+0000","last_peered":"2024-09-16T04:21:16.871100+0000","last_clean":"2024-09-16T04:21:16.871100+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T04:21:16.871100+0000","last_undegraded":"2024-09-16T04:21:16.871100+0000","last_fullsized":"2024-09-16T04:21:16.871100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865432+0000","last_change":"2024-09-16T03:50:58.548783+0000","last_active":"2024-09-16T04:21:14.865432+0000","last_peered":"2024-09-16T04:21:14.865432+0000","last_clean":"2024-09-16T04:21:14.865432+0000","last_became_active":"2024-09-16T03:50:58.545532+0000","last_became_peered":"2024-09-16T03:50:58.545532+0000","last_unstale":"2024-09-16T04:21:14.865432+0000","last_undegraded":"2024-09-16T04:21:14.865432+0000","last_fullsized":"2024-09-16T04:21:14.865432+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:46.492270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":1458,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878077+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T04:21:16.878077+0000","last_peered":"2024-09-16T04:21:16.878077+0000","last_clean":"2024-09-16T04:21:16.878077+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T04:21:16.878077+0000","last_undegraded":"2024-09-16T04:21:16.878077+0000","last_fullsized":"2024-09-16T04:21:16.878077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865416+0000","last_change":"2024-09-16T03:50:58.560987+0000","last_active":"2024-09-16T04:21:14.865416+0000","last_peered":"2024-09-16T04:21:14.865416+0000","last_clean":"2024-09-16T04:21:14.865416+0000","last_became_active":"2024-09-16T03:50:58.555850+0000","last_became_peered":"2024-09-16T03:50:58.555850+0000","last_unstale":"2024-09-16T04:21:14.865416+0000","last_undegraded":"2024-09-16T04:21:14.865416+0000","last_fullsized":"2024-09-16T04:21:14.865416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:43.936131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1459,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879705+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T04:21:15.879705+0000","last_peered":"2024-09-16T04:21:15.879705+0000","last_clean":"2024-09-16T04:21:15.879705+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T04:21:15.879705+0000","last_undegraded":"2024-09-16T04:21:15.879705+0000","last_fullsized":"2024-09-16T04:21:15.879705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:20.389723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1450,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856298+0000","last_change":"2024-09-16T03:50:58.561772+0000","last_active":"2024-09-16T04:21:12.856298+0000","last_peered":"2024-09-16T04:21:12.856298+0000","last_clean":"2024-09-16T04:21:12.856298+0000","last_became_active":"2024-09-16T03:50:58.561522+0000","last_became_peered":"2024-09-16T03:50:58.561522+0000","last_unstale":"2024-09-16T04:21:12.856298+0000","last_undegraded":"2024-09-16T04:21:12.856298+0000","last_fullsized":"2024-09-16T04:21:12.856298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:46.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871185+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T04:21:16.871185+0000","last_peered":"2024-09-16T04:21:16.871185+0000","last_clean":"2024-09-16T04:21:16.871185+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T04:21:16.871185+0000","last_undegraded":"2024-09-16T04:21:16.871185+0000","last_fullsized":"2024-09-16T04:21:16.871185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"24'7","reported_seq":1474,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872634+0000","last_change":"2024-09-16T03:50:58.476506+0000","last_active":"2024-09-16T04:21:16.872634+0000","last_peered":"2024-09-16T04:21:16.872634+0000","last_clean":"2024-09-16T04:21:16.872634+0000","last_became_active":"2024-09-16T03:50:58.476200+0000","last_became_peered":"2024-09-16T03:50:58.476200+0000","last_unstale":"2024-09-16T04:21:16.872634+0000","last_undegraded":"2024-09-16T04:21:16.872634+0000","last_fullsized":"2024-09-16T04:21:16.872634+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:05.408395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1458,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876027+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T04:21:15.876027+0000","last_peered":"2024-09-16T04:21:15.876027+0000","last_clean":"2024-09-16T04:21:15.876027+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T04:21:15.876027+0000","last_undegraded":"2024-09-16T04:21:15.876027+0000","last_fullsized":"2024-09-16T04:21:15.876027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:00.954047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1463,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872030+0000","last_change":"2024-09-16T03:50:58.522695+0000","last_active":"2024-09-16T04:21:15.872030+0000","last_peered":"2024-09-16T04:21:15.872030+0000","last_clean":"2024-09-16T04:21:15.872030+0000","last_became_active":"2024-09-16T03:50:58.522014+0000","last_became_peered":"2024-09-16T03:50:58.522014+0000","last_unstale":"2024-09-16T04:21:15.872030+0000","last_undegraded":"2024-09-16T04:21:15.872030+0000","last_fullsized":"2024-09-16T04:21:15.872030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:38.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1463,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856460+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T04:21:12.856460+0000","last_peered":"2024-09-16T04:21:12.856460+0000","last_clean":"2024-09-16T04:21:12.856460+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T04:21:12.856460+0000","last_undegraded":"2024-09-16T04:21:12.856460+0000","last_fullsized":"2024-09-16T04:21:12.856460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.877443+0000","last_change":"2024-09-16T03:50:58.520798+0000","last_active":"2024-09-16T04:21:15.877443+0000","last_peered":"2024-09-16T04:21:15.877443+0000","last_clean":"2024-09-16T04:21:15.877443+0000","last_became_active":"2024-09-16T03:50:58.519191+0000","last_became_peered":"2024-09-16T03:50:58.519191+0000","last_unstale":"2024-09-16T04:21:15.877443+0000","last_undegraded":"2024-09-16T04:21:15.877443+0000","last_fullsized":"2024-09-16T04:21:15.877443+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:15.389894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872620+0000","last_change":"2024-09-16T03:50:58.547695+0000","last_active":"2024-09-16T04:21:15.872620+0000","last_peered":"2024-09-16T04:21:15.872620+0000","last_clean":"2024-09-16T04:21:15.872620+0000","last_became_active":"2024-09-16T03:50:58.546981+0000","last_became_peered":"2024-09-16T03:50:58.546981+0000","last_unstale":"2024-09-16T04:21:15.872620+0000","last_undegraded":"2024-09-16T04:21:15.872620+0000","last_fullsized":"2024-09-16T04:21:15.872620+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:29.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870965+0000","last_change":"2024-09-16T03:50:58.542578+0000","last_active":"2024-09-16T04:21:16.870965+0000","last_peered":"2024-09-16T04:21:16.870965+0000","last_clean":"2024-09-16T04:21:16.870965+0000","last_became_active":"2024-09-16T03:50:58.541780+0000","last_became_peered":"2024-09-16T03:50:58.541780+0000","last_unstale":"2024-09-16T04:21:16.870965+0000","last_undegraded":"2024-09-16T04:21:16.870965+0000","last_fullsized":"2024-09-16T04:21:16.870965+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:34.596173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1448,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856489+0000","last_change":"2024-09-16T03:50:58.542966+0000","last_active":"2024-09-16T04:21:12.856489+0000","last_peered":"2024-09-16T04:21:12.856489+0000","last_clean":"2024-09-16T04:21:12.856489+0000","last_became_active":"2024-09-16T03:50:58.542726+0000","last_became_peered":"2024-09-16T03:50:58.542726+0000","last_unstale":"2024-09-16T04:21:12.856489+0000","last_undegraded":"2024-09-16T04:21:12.856489+0000","last_fullsized":"2024-09-16T04:21:12.856489+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:11.421727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1464,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865544+0000","last_change":"2024-09-16T03:50:58.558551+0000","last_active":"2024-09-16T04:21:14.865544+0000","last_peered":"2024-09-16T04:21:14.865544+0000","last_clean":"2024-09-16T04:21:14.865544+0000","last_became_active":"2024-09-16T03:50:58.557890+0000","last_became_peered":"2024-09-16T03:50:58.557890+0000","last_unstale":"2024-09-16T04:21:14.865544+0000","last_undegraded":"2024-09-16T04:21:14.865544+0000","last_fullsized":"2024-09-16T04:21:14.865544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:13.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1450,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856392+0000","last_change":"2024-09-16T03:50:58.545310+0000","last_active":"2024-09-16T04:21:12.856392+0000","last_peered":"2024-09-16T04:21:12.856392+0000","last_clean":"2024-09-16T04:21:12.856392+0000","last_became_active":"2024-09-16T03:50:58.544536+0000","last_became_peered":"2024-09-16T03:50:58.544536+0000","last_unstale":"2024-09-16T04:21:12.856392+0000","last_undegraded":"2024-09-16T04:21:12.856392+0000","last_fullsized":"2024-09-16T04:21:12.856392+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:34:15.701626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872785+0000","last_change":"2024-09-16T03:50:58.570630+0000","last_active":"2024-09-16T04:21:16.872785+0000","last_peered":"2024-09-16T04:21:16.872785+0000","last_clean":"2024-09-16T04:21:16.872785+0000","last_became_active":"2024-09-16T03:50:58.570205+0000","last_became_peered":"2024-09-16T03:50:58.570205+0000","last_unstale":"2024-09-16T04:21:16.872785+0000","last_undegraded":"2024-09-16T04:21:16.872785+0000","last_fullsized":"2024-09-16T04:21:16.872785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:23.406420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876760+0000","last_change":"2024-09-16T03:50:58.545353+0000","last_active":"2024-09-16T04:21:15.876760+0000","last_peered":"2024-09-16T04:21:15.876760+0000","last_clean":"2024-09-16T04:21:15.876760+0000","last_became_active":"2024-09-16T03:50:58.543727+0000","last_became_peered":"2024-09-16T03:50:58.543727+0000","last_unstale":"2024-09-16T04:21:15.876760+0000","last_undegraded":"2024-09-16T04:21:15.876760+0000","last_fullsized":"2024-09-16T04:21:15.876760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:41.271697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871001+0000","last_change":"2024-09-16T03:50:58.550926+0000","last_active":"2024-09-16T04:21:16.871001+0000","last_peered":"2024-09-16T04:21:16.871001+0000","last_clean":"2024-09-16T04:21:16.871001+0000","last_became_active":"2024-09-16T03:50:58.549440+0000","last_became_peered":"2024-09-16T03:50:58.549440+0000","last_unstale":"2024-09-16T04:21:16.871001+0000","last_undegraded":"2024-09-16T04:21:16.871001+0000","last_fullsized":"2024-09-16T04:21:16.871001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:58:40.013005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878320+0000","last_change":"2024-09-16T03:50:58.483761+0000","last_active":"2024-09-16T04:21:16.878320+0000","last_peered":"2024-09-16T04:21:16.878320+0000","last_clean":"2024-09-16T04:21:16.878320+0000","last_became_active":"2024-09-16T03:50:58.483098+0000","last_became_peered":"2024-09-16T03:50:58.483098+0000","last_unstale":"2024-09-16T04:21:16.878320+0000","last_undegraded":"2024-09-16T04:21:16.878320+0000","last_fullsized":"2024-09-16T04:21:16.878320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:30.723960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872415+0000","last_change":"2024-09-16T03:50:58.524457+0000","last_active":"2024-09-16T04:21:16.872415+0000","last_peered":"2024-09-16T04:21:16.872415+0000","last_clean":"2024-09-16T04:21:16.872415+0000","last_became_active":"2024-09-16T03:50:58.524223+0000","last_became_peered":"2024-09-16T03:50:58.524223+0000","last_unstale":"2024-09-16T04:21:16.872415+0000","last_undegraded":"2024-09-16T04:21:16.872415+0000","last_fullsized":"2024-09-16T04:21:16.872415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:41.294641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.875858+0000","last_change":"2024-09-16T03:50:58.520158+0000","last_active":"2024-09-16T04:21:15.875858+0000","last_peered":"2024-09-16T04:21:15.875858+0000","last_clean":"2024-09-16T04:21:15.875858+0000","last_became_active":"2024-09-16T03:50:58.519543+0000","last_became_peered":"2024-09-16T03:50:58.519543+0000","last_unstale":"2024-09-16T04:21:15.875858+0000","last_undegraded":"2024-09-16T04:21:15.875858+0000","last_fullsized":"2024-09-16T04:21:15.875858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:48.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509823,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68480,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303360,"statfs":{"total":96636764160,"available":96566640640,"internally_reserved":0,"allocated":4923392,"data_stored":2365351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67464,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304376,"statfs":{"total":96636764160,"available":96567681024,"internally_reserved":0,"allocated":4071424,"data_stored":1806610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542526,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68328,"kb_used_data":4840,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303512,"statfs":{"total":96636764160,"available":96566796288,"internally_reserved":0,"allocated":4956160,"data_stored":2572914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79928,"kb_used_data":4144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291912,"statfs":{"total":96636764160,"available":96554917888,"internally_reserved":0,"allocated":4243456,"data_stored":1927659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542525,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71604,"kb_used_data":4324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300236,"statfs":{"total":96636764160,"available":96563441664,"internally_reserved":0,"allocated":4427776,"data_stored":2043991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76132,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295708,"statfs":{"total":96636764160,"available":96558804992,"internally_reserved":0,"allocated":4239360,"data_stored":1927640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80112,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291728,"statfs":{"total":96636764160,"available":96554729472,"internally_reserved":0,"allocated":4431872,"data_stored":2044009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76620,"kb_used_data":4996,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295220,"statfs":{"total":96636764160,"available":96558305280,"internally_reserved":0,"allocated":5115904,"data_stored":2675247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:21:18.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:21:18.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.880+0000 7f553b3e4640 10 monclient: get_monmap_and_config 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.880+0000 7f553b3e4640 10 monclient: build_initial_monmap 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.880+0000 7f553b3e4640 10 monclient: monmap: 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:18.880882+0000 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:18.880882+0000 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:18.882 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:18.883 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 1 Processor -- start 2024-09-16T04:21:18.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 1 -- start start 2024-09-16T04:21:18.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient: init 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55340b5650 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55340b3120 0x7f5534146500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-a con 0x7f55340b3120 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5534146a40 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55340aae40 con 0x7f55340b5650 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.881+0000 7f553b3e4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5534140b20 con 0x7f5534146a40 2024-09-16T04:21:18.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553b3e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553b3e4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f55340b5a30 con 0x7f55340b3120 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553b3e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553b3e4640 10 monclient(hunting): authenticate will time out at 2637.543019s 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55340b3120 0x7f5534146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f5539180640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f55340b5650 0x7f5534140410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42436/0 (socket says 172.21.15.70:42436) 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f5539180640 1 -- 172.21.15.70:0/2401638551 learned_addr learned my addr 172.21.15.70:0/2401638551 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): get_auth_request con 0x7f55340b3120 auth_method 0 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient(hunting): handle_auth_done global_id 26709 payload 274 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 1 -- 172.21.15.70:0/2401638551 >> v1:172.21.15.163:6789/0 conn(0x7f5534146a40 legacy=0x7f553414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 1 -- 172.21.15.70:0/2401638551 >> v1:172.21.15.70:6789/0 conn(0x7f55340b5650 legacy=0x7f5534140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient: _finish_hunting 0 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 1 monclient: found mon.noname-a 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f552bfff640 1 -- 172.21.15.70:0/2401638551 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256231433 0 0) 0x7f55340aae40 con 0x7f55340b5650 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f553897f640 1 -- 172.21.15.70:0/2401638551 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55340b5cd0 con 0x7f55340b3120 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.882+0000 7f552bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553897f640 1 --2- 172.21.15.70:0/2401638551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55340b3120 0x7f5534146500 secure :-1 s=READY pgs=5731 cs=0 l=1 rev1=1 crypto rx=0x7f55240097a0 tx=0x7f552402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97db8afd3d984d17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 1 -- 172.21.15.70:0/2401638551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524037070 con 0x7f55340b3120 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: dump: 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.897 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: _finish_auth 0 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 1 -- 172.21.15.70:0/2401638551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5524032440 con 0x7f55340b3120 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 5 monclient: authenticate success, global_id 26709 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 1 -- 172.21.15.70:0/2401638551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524032810 con 0x7f55340b3120 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 4 set_mon_vals no callback set 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f552bfff640 10 monclient: dump: 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 10 monclient: shutdown 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:18.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 1 -- 172.21.15.70:0/2401638551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55340b3120 msgr2=0x7f5534146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 1 --2- 172.21.15.70:0/2401638551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55340b3120 0x7f5534146500 secure :-1 s=READY pgs=5731 cs=0 l=1 rev1=1 crypto rx=0x7f55240097a0 tx=0x7f552402a1a0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 1 -- 172.21.15.70:0/2401638551 shutdown_connections 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.883+0000 7f553b3e4640 1 --2- 172.21.15.70:0/2401638551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55340b3120 0x7f5534146500 unknown :-1 s=CLOSED pgs=5731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 1 -- 172.21.15.70:0/2401638551 >> 172.21.15.70:0/2401638551 conn(0x7f55340679c0 msgr2=0x7f5534067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 1 -- 172.21.15.70:0/2401638551 shutdown_connections 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 1 -- 172.21.15.70:0/2401638551 wait complete. 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 10 monclient: build_initial_monmap 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 10 monclient: monmap: 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 1 librados: starting msgr at 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.884+0000 7f553b3e4640 1 librados: starting objecter 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 Processor -- start 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 -- start start 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 librados: setting wanted keys 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 librados: calling monclient init 2024-09-16T04:21:18.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient: init 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-c con 0x7f55340b3120 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55340b5650 0x7f5534143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-b con 0x7f55340b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5534146a40 0x7f55340ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): picked mon.noname-a con 0x7f5534146a40 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5534146fd0 con 0x7f5534146a40 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5534146e50 con 0x7f55340b5650 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f5534147150 con 0x7f55340b3120 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553b3e4640 10 monclient(hunting): authenticate will time out at 2637.546598s 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54706/0 (socket says 172.21.15.70:54706) 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 -- 172.21.15.70:0/1841578423 learned_addr learned my addr 172.21.15.70:0/1841578423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): get_auth_request con 0x7f55340b3120 auth_method 0 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5534146a40 0x7f55340ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): get_auth_request con 0x7f5534146a40 auth_method 0 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539981640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f553897f640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55340b5650 0x7f5534143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient(hunting): handle_auth_done global_id 20951 payload 824 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 -- 172.21.15.70:0/1841578423 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55340b5650 msgr2=0x7f5534143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55340b5650 0x7f5534143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 -- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5534146a40 msgr2=0x7f55340ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5534146a40 0x7f55340ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 10 monclient: _finish_hunting 0 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 1 monclient: found mon.noname-c 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.885+0000 7f5539180640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:18.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5539180640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5539180640 1 -- 172.21.15.70:0/1841578423 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55340ae1b0 con 0x7f55340b3120 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5539180640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f553000c4a0 tx=0x7f553000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 client.?.objecter ms_handle_connect 0x7f55340b3120 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530018070 con 0x7f55340b3120 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: dump: 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.906 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55300074e0 con 0x7f55340b3120 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 5 monclient: authenticate success, global_id 20951 2024-09-16T04:21:18.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 10 monclient: _renew_subs 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55340ae490 con 0x7f55340b3120 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530004a20 con 0x7f55340b3120 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f5529ffb640 10 monclient: dump: 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:18.908 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f552b7fe640 4 set_mon_vals no callback set 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 10 client.20951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 10 monclient: _renew_subs 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55341dbb10 con 0x7f55340b3120 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.886+0000 7f553b3e4640 1 librados: init done 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.887+0000 7f553b3e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.887+0000 7f553b3e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.887+0000 7f553b3e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:18.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.887+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5534146500 con 0x7f55340b3120 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.887+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f553001b070 con 0x7f55340b3120 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 1 --2- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 0x7f55180700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f553897f640 1 --2- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 0x7f55180700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f553008a4b0 con 0x7f55340b3120 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 10 client.20951.objecter ms_dispatch 0x7f5534000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 3 client.20951.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 3 client.20951.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f553897f640 10 monclient: get_auth_request con 0x7f551806dc00 auth_method 0 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 20 client.20951.objecter dump_active .. 0 homeless 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f5529ffb640 10 client.20951.objecter ms_handle_connect 0x7f551806dc00 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.888+0000 7f553897f640 1 --2- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 0x7f55180700c0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f5524034000 tx=0x7f552402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.892+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5530058820 con 0x7f55340b3120 2024-09-16T04:21:18.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.892+0000 7f5529ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:18.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:18.892+0000 7f5529ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:19.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.060+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f55341437d0 con 0x7f551806dc00 2024-09-16T04:21:19.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.063+0000 7f5529ffb640 1 -- 172.21.15.70:0/1841578423 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116304 (secure 0 0 0) 0x7f55341437d0 con 0x7f551806dc00 2024-09-16T04:21:19.065 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:19.065 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T04:21:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 10 librados: watch_flush enter 2024-09-16T04:21:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 10 librados: watch_flush exit 2024-09-16T04:21:19.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 20 client.20951.objecter shutdown clearing up homeless session... 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 10 client.20951.objecter successfully canceled tick 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 msgr2=0x7f55180700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 0x7f55180700c0 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f5524034000 tx=0x7f552402e000 comp rx=0 tx=0).stop 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 10 monclient: shutdown 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 msgr2=0x7f55341ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f553000c4a0 tx=0x7f553000c960 comp rx=0 tx=0).stop 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 shutdown_connections 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> v2:172.21.15.70:6804/198622594 conn(0x7f551806dc00 0x7f55180700c0 unknown :-1 s=CLOSED pgs=4025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5534146a40 0x7f55340ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55340b5650 0x7f5534143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 --2- 172.21.15.70:0/1841578423 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f55340b3120 0x7f55341ce7f0 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 >> 172.21.15.70:0/1841578423 conn(0x7f55340679c0 msgr2=0x7f55340b9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 shutdown_connections 2024-09-16T04:21:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 -- 172.21.15.70:0/1841578423 wait complete. 2024-09-16T04:21:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.065+0000 7f553b3e4640 1 librados: shutdown 2024-09-16T04:21:19.077 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1693,"stamp":"2024-09-16T04:21:17.878077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":588668,"kb_used_data":35556,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754386052,"statfs":{"total":773094113280,"available":772491317248,"internally_reserved":0,"allocated":36409344,"data_stored":17363421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":563793458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020319"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878232+0000","last_change":"2024-09-16T03:50:58.527921+0000","last_active":"2024-09-16T04:21:16.878232+0000","last_peered":"2024-09-16T04:21:16.878232+0000","last_clean":"2024-09-16T04:21:16.878232+0000","last_became_active":"2024-09-16T03:50:58.527591+0000","last_became_peered":"2024-09-16T03:50:58.527591+0000","last_unstale":"2024-09-16T04:21:16.878232+0000","last_undegraded":"2024-09-16T04:21:16.878232+0000","last_fullsized":"2024-09-16T04:21:16.878232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.155864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872367+0000","last_change":"2024-09-16T03:50:58.566215+0000","last_active":"2024-09-16T04:21:15.872367+0000","last_peered":"2024-09-16T04:21:15.872367+0000","last_clean":"2024-09-16T04:21:15.872367+0000","last_became_active":"2024-09-16T03:50:58.565169+0000","last_became_peered":"2024-09-16T03:50:58.565169+0000","last_unstale":"2024-09-16T04:21:15.872367+0000","last_undegraded":"2024-09-16T04:21:15.872367+0000","last_fullsized":"2024-09-16T04:21:15.872367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:31.209733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872690+0000","last_change":"2024-09-16T03:50:58.521189+0000","last_active":"2024-09-16T04:21:15.872690+0000","last_peered":"2024-09-16T04:21:15.872690+0000","last_clean":"2024-09-16T04:21:15.872690+0000","last_became_active":"2024-09-16T03:50:58.520920+0000","last_became_peered":"2024-09-16T03:50:58.520920+0000","last_unstale":"2024-09-16T04:21:15.872690+0000","last_undegraded":"2024-09-16T04:21:15.872690+0000","last_fullsized":"2024-09-16T04:21:15.872690+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:15.187842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878466+0000","last_change":"2024-09-16T03:50:58.529897+0000","last_active":"2024-09-16T04:21:16.878466+0000","last_peered":"2024-09-16T04:21:16.878466+0000","last_clean":"2024-09-16T04:21:16.878466+0000","last_became_active":"2024-09-16T03:50:58.529616+0000","last_became_peered":"2024-09-16T03:50:58.529616+0000","last_unstale":"2024-09-16T04:21:16.878466+0000","last_undegraded":"2024-09-16T04:21:16.878466+0000","last_fullsized":"2024-09-16T04:21:16.878466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:40.902750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870995+0000","last_change":"2024-09-16T03:50:58.547657+0000","last_active":"2024-09-16T04:21:16.870995+0000","last_peered":"2024-09-16T04:21:16.870995+0000","last_clean":"2024-09-16T04:21:16.870995+0000","last_became_active":"2024-09-16T03:50:58.547418+0000","last_became_peered":"2024-09-16T03:50:58.547418+0000","last_unstale":"2024-09-16T04:21:16.870995+0000","last_undegraded":"2024-09-16T04:21:16.870995+0000","last_fullsized":"2024-09-16T04:21:16.870995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:45.151909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871082+0000","last_change":"2024-09-16T03:50:58.563918+0000","last_active":"2024-09-16T04:21:16.871082+0000","last_peered":"2024-09-16T04:21:16.871082+0000","last_clean":"2024-09-16T04:21:16.871082+0000","last_became_active":"2024-09-16T03:50:58.563681+0000","last_became_peered":"2024-09-16T03:50:58.563681+0000","last_unstale":"2024-09-16T04:21:16.871082+0000","last_undegraded":"2024-09-16T04:21:16.871082+0000","last_fullsized":"2024-09-16T04:21:16.871082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:01.363054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871296+0000","last_change":"2024-09-16T03:50:58.563162+0000","last_active":"2024-09-16T04:21:16.871296+0000","last_peered":"2024-09-16T04:21:16.871296+0000","last_clean":"2024-09-16T04:21:16.871296+0000","last_became_active":"2024-09-16T03:50:58.562917+0000","last_became_peered":"2024-09-16T03:50:58.562917+0000","last_unstale":"2024-09-16T04:21:16.871296+0000","last_undegraded":"2024-09-16T04:21:16.871296+0000","last_fullsized":"2024-09-16T04:21:16.871296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:20.514696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871351+0000","last_change":"2024-09-16T03:50:58.552412+0000","last_active":"2024-09-16T04:21:16.871351+0000","last_peered":"2024-09-16T04:21:16.871351+0000","last_clean":"2024-09-16T04:21:16.871351+0000","last_became_active":"2024-09-16T03:50:58.552137+0000","last_became_peered":"2024-09-16T03:50:58.552137+0000","last_unstale":"2024-09-16T04:21:16.871351+0000","last_undegraded":"2024-09-16T04:21:16.871351+0000","last_fullsized":"2024-09-16T04:21:16.871351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:13.498957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871506+0000","last_change":"2024-09-16T03:50:58.546028+0000","last_active":"2024-09-16T04:21:16.871506+0000","last_peered":"2024-09-16T04:21:16.871506+0000","last_clean":"2024-09-16T04:21:16.871506+0000","last_became_active":"2024-09-16T03:50:58.545776+0000","last_became_peered":"2024-09-16T03:50:58.545776+0000","last_unstale":"2024-09-16T04:21:16.871506+0000","last_undegraded":"2024-09-16T04:21:16.871506+0000","last_fullsized":"2024-09-16T04:21:16.871506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:03.340804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1442,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879550+0000","last_change":"2024-09-16T03:50:58.540624+0000","last_active":"2024-09-16T04:21:15.879550+0000","last_peered":"2024-09-16T04:21:15.879550+0000","last_clean":"2024-09-16T04:21:15.879550+0000","last_became_active":"2024-09-16T03:50:58.540379+0000","last_became_peered":"2024-09-16T03:50:58.540379+0000","last_unstale":"2024-09-16T04:21:15.879550+0000","last_undegraded":"2024-09-16T04:21:15.879550+0000","last_fullsized":"2024-09-16T04:21:15.879550+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:12.307201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879223+0000","last_change":"2024-09-16T03:50:58.480959+0000","last_active":"2024-09-16T04:21:16.879223+0000","last_peered":"2024-09-16T04:21:16.879223+0000","last_clean":"2024-09-16T04:21:16.879223+0000","last_became_active":"2024-09-16T03:50:58.480684+0000","last_became_peered":"2024-09-16T03:50:58.480684+0000","last_unstale":"2024-09-16T04:21:16.879223+0000","last_undegraded":"2024-09-16T04:21:16.879223+0000","last_fullsized":"2024-09-16T04:21:16.879223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:38.866093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1460,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872748+0000","last_change":"2024-09-16T03:50:58.508324+0000","last_active":"2024-09-16T04:21:16.872748+0000","last_peered":"2024-09-16T04:21:16.872748+0000","last_clean":"2024-09-16T04:21:16.872748+0000","last_became_active":"2024-09-16T03:50:58.507923+0000","last_became_peered":"2024-09-16T03:50:58.507923+0000","last_unstale":"2024-09-16T04:21:16.872748+0000","last_undegraded":"2024-09-16T04:21:16.872748+0000","last_fullsized":"2024-09-16T04:21:16.872748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:50.490187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871014+0000","last_change":"2024-09-16T03:50:58.539556+0000","last_active":"2024-09-16T04:21:16.871014+0000","last_peered":"2024-09-16T04:21:16.871014+0000","last_clean":"2024-09-16T04:21:16.871014+0000","last_became_active":"2024-09-16T03:50:58.539098+0000","last_became_peered":"2024-09-16T03:50:58.539098+0000","last_unstale":"2024-09-16T04:21:16.871014+0000","last_undegraded":"2024-09-16T04:21:16.871014+0000","last_fullsized":"2024-09-16T04:21:16.871014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:34.294689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":1616,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871130+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T04:21:16.871130+0000","last_peered":"2024-09-16T04:21:16.871130+0000","last_clean":"2024-09-16T04:21:16.871130+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T04:21:16.871130+0000","last_undegraded":"2024-09-16T04:21:16.871130+0000","last_fullsized":"2024-09-16T04:21:16.871130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":1457,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878405+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T04:21:16.878405+0000","last_peered":"2024-09-16T04:21:16.878405+0000","last_clean":"2024-09-16T04:21:16.878405+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T04:21:16.878405+0000","last_undegraded":"2024-09-16T04:21:16.878405+0000","last_fullsized":"2024-09-16T04:21:16.878405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879231+0000","last_change":"2024-09-16T03:50:58.527360+0000","last_active":"2024-09-16T04:21:16.879231+0000","last_peered":"2024-09-16T04:21:16.879231+0000","last_clean":"2024-09-16T04:21:16.879231+0000","last_became_active":"2024-09-16T03:50:58.527023+0000","last_became_peered":"2024-09-16T03:50:58.527023+0000","last_unstale":"2024-09-16T04:21:16.879231+0000","last_undegraded":"2024-09-16T04:21:16.879231+0000","last_fullsized":"2024-09-16T04:21:16.879231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:42.928632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871059+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T04:21:16.871059+0000","last_peered":"2024-09-16T04:21:16.871059+0000","last_clean":"2024-09-16T04:21:16.871059+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T04:21:16.871059+0000","last_undegraded":"2024-09-16T04:21:16.871059+0000","last_fullsized":"2024-09-16T04:21:16.871059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871813+0000","last_change":"2024-09-16T03:50:58.528877+0000","last_active":"2024-09-16T04:21:16.871813+0000","last_peered":"2024-09-16T04:21:16.871813+0000","last_clean":"2024-09-16T04:21:16.871813+0000","last_became_active":"2024-09-16T03:50:58.528663+0000","last_became_peered":"2024-09-16T03:50:58.528663+0000","last_unstale":"2024-09-16T04:21:16.871813+0000","last_undegraded":"2024-09-16T04:21:16.871813+0000","last_fullsized":"2024-09-16T04:21:16.871813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.620543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1463,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871100+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T04:21:16.871100+0000","last_peered":"2024-09-16T04:21:16.871100+0000","last_clean":"2024-09-16T04:21:16.871100+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T04:21:16.871100+0000","last_undegraded":"2024-09-16T04:21:16.871100+0000","last_fullsized":"2024-09-16T04:21:16.871100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865432+0000","last_change":"2024-09-16T03:50:58.548783+0000","last_active":"2024-09-16T04:21:14.865432+0000","last_peered":"2024-09-16T04:21:14.865432+0000","last_clean":"2024-09-16T04:21:14.865432+0000","last_became_active":"2024-09-16T03:50:58.545532+0000","last_became_peered":"2024-09-16T03:50:58.545532+0000","last_unstale":"2024-09-16T04:21:14.865432+0000","last_undegraded":"2024-09-16T04:21:14.865432+0000","last_fullsized":"2024-09-16T04:21:14.865432+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:46.492270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":1458,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878077+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T04:21:16.878077+0000","last_peered":"2024-09-16T04:21:16.878077+0000","last_clean":"2024-09-16T04:21:16.878077+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T04:21:16.878077+0000","last_undegraded":"2024-09-16T04:21:16.878077+0000","last_fullsized":"2024-09-16T04:21:16.878077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865416+0000","last_change":"2024-09-16T03:50:58.560987+0000","last_active":"2024-09-16T04:21:14.865416+0000","last_peered":"2024-09-16T04:21:14.865416+0000","last_clean":"2024-09-16T04:21:14.865416+0000","last_became_active":"2024-09-16T03:50:58.555850+0000","last_became_peered":"2024-09-16T03:50:58.555850+0000","last_unstale":"2024-09-16T04:21:14.865416+0000","last_undegraded":"2024-09-16T04:21:14.865416+0000","last_fullsized":"2024-09-16T04:21:14.865416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:43.936131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1459,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879705+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T04:21:15.879705+0000","last_peered":"2024-09-16T04:21:15.879705+0000","last_clean":"2024-09-16T04:21:15.879705+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T04:21:15.879705+0000","last_undegraded":"2024-09-16T04:21:15.879705+0000","last_fullsized":"2024-09-16T04:21:15.879705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:20.389723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1450,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856298+0000","last_change":"2024-09-16T03:50:58.561772+0000","last_active":"2024-09-16T04:21:12.856298+0000","last_peered":"2024-09-16T04:21:12.856298+0000","last_clean":"2024-09-16T04:21:12.856298+0000","last_became_active":"2024-09-16T03:50:58.561522+0000","last_became_peered":"2024-09-16T03:50:58.561522+0000","last_unstale":"2024-09-16T04:21:12.856298+0000","last_undegraded":"2024-09-16T04:21:12.856298+0000","last_fullsized":"2024-09-16T04:21:12.856298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:46.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871185+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T04:21:16.871185+0000","last_peered":"2024-09-16T04:21:16.871185+0000","last_clean":"2024-09-16T04:21:16.871185+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T04:21:16.871185+0000","last_undegraded":"2024-09-16T04:21:16.871185+0000","last_fullsized":"2024-09-16T04:21:16.871185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"24'7","reported_seq":1474,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872634+0000","last_change":"2024-09-16T03:50:58.476506+0000","last_active":"2024-09-16T04:21:16.872634+0000","last_peered":"2024-09-16T04:21:16.872634+0000","last_clean":"2024-09-16T04:21:16.872634+0000","last_became_active":"2024-09-16T03:50:58.476200+0000","last_became_peered":"2024-09-16T03:50:58.476200+0000","last_unstale":"2024-09-16T04:21:16.872634+0000","last_undegraded":"2024-09-16T04:21:16.872634+0000","last_fullsized":"2024-09-16T04:21:16.872634+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:05.408395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1458,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876027+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T04:21:15.876027+0000","last_peered":"2024-09-16T04:21:15.876027+0000","last_clean":"2024-09-16T04:21:15.876027+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T04:21:15.876027+0000","last_undegraded":"2024-09-16T04:21:15.876027+0000","last_fullsized":"2024-09-16T04:21:15.876027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:00.954047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1463,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872030+0000","last_change":"2024-09-16T03:50:58.522695+0000","last_active":"2024-09-16T04:21:15.872030+0000","last_peered":"2024-09-16T04:21:15.872030+0000","last_clean":"2024-09-16T04:21:15.872030+0000","last_became_active":"2024-09-16T03:50:58.522014+0000","last_became_peered":"2024-09-16T03:50:58.522014+0000","last_unstale":"2024-09-16T04:21:15.872030+0000","last_undegraded":"2024-09-16T04:21:15.872030+0000","last_fullsized":"2024-09-16T04:21:15.872030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:38.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1463,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856460+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T04:21:12.856460+0000","last_peered":"2024-09-16T04:21:12.856460+0000","last_clean":"2024-09-16T04:21:12.856460+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T04:21:12.856460+0000","last_undegraded":"2024-09-16T04:21:12.856460+0000","last_fullsized":"2024-09-16T04:21:12.856460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.877443+0000","last_change":"2024-09-16T03:50:58.520798+0000","last_active":"2024-09-16T04:21:15.877443+0000","last_peered":"2024-09-16T04:21:15.877443+0000","last_clean":"2024-09-16T04:21:15.877443+0000","last_became_active":"2024-09-16T03:50:58.519191+0000","last_became_peered":"2024-09-16T03:50:58.519191+0000","last_unstale":"2024-09-16T04:21:15.877443+0000","last_undegraded":"2024-09-16T04:21:15.877443+0000","last_fullsized":"2024-09-16T04:21:15.877443+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:15.389894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872620+0000","last_change":"2024-09-16T03:50:58.547695+0000","last_active":"2024-09-16T04:21:15.872620+0000","last_peered":"2024-09-16T04:21:15.872620+0000","last_clean":"2024-09-16T04:21:15.872620+0000","last_became_active":"2024-09-16T03:50:58.546981+0000","last_became_peered":"2024-09-16T03:50:58.546981+0000","last_unstale":"2024-09-16T04:21:15.872620+0000","last_undegraded":"2024-09-16T04:21:15.872620+0000","last_fullsized":"2024-09-16T04:21:15.872620+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:29.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870965+0000","last_change":"2024-09-16T03:50:58.542578+0000","last_active":"2024-09-16T04:21:16.870965+0000","last_peered":"2024-09-16T04:21:16.870965+0000","last_clean":"2024-09-16T04:21:16.870965+0000","last_became_active":"2024-09-16T03:50:58.541780+0000","last_became_peered":"2024-09-16T03:50:58.541780+0000","last_unstale":"2024-09-16T04:21:16.870965+0000","last_undegraded":"2024-09-16T04:21:16.870965+0000","last_fullsized":"2024-09-16T04:21:16.870965+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:34.596173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1448,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856489+0000","last_change":"2024-09-16T03:50:58.542966+0000","last_active":"2024-09-16T04:21:12.856489+0000","last_peered":"2024-09-16T04:21:12.856489+0000","last_clean":"2024-09-16T04:21:12.856489+0000","last_became_active":"2024-09-16T03:50:58.542726+0000","last_became_peered":"2024-09-16T03:50:58.542726+0000","last_unstale":"2024-09-16T04:21:12.856489+0000","last_undegraded":"2024-09-16T04:21:12.856489+0000","last_fullsized":"2024-09-16T04:21:12.856489+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:11.421727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1464,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865544+0000","last_change":"2024-09-16T03:50:58.558551+0000","last_active":"2024-09-16T04:21:14.865544+0000","last_peered":"2024-09-16T04:21:14.865544+0000","last_clean":"2024-09-16T04:21:14.865544+0000","last_became_active":"2024-09-16T03:50:58.557890+0000","last_became_peered":"2024-09-16T03:50:58.557890+0000","last_unstale":"2024-09-16T04:21:14.865544+0000","last_undegraded":"2024-09-16T04:21:14.865544+0000","last_fullsized":"2024-09-16T04:21:14.865544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:13.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1450,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-09-16T04:21:12.856392+0000","last_change":"2024-09-16T03:50:58.545310+0000","last_active":"2024-09-16T04:21:12.856392+0000","last_peered":"2024-09-16T04:21:12.856392+0000","last_clean":"2024-09-16T04:21:12.856392+0000","last_became_active":"2024-09-16T03:50:58.544536+0000","last_became_peered":"2024-09-16T03:50:58.544536+0000","last_unstale":"2024-09-16T04:21:12.856392+0000","last_undegraded":"2024-09-16T04:21:12.856392+0000","last_fullsized":"2024-09-16T04:21:12.856392+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:34:15.701626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872785+0000","last_change":"2024-09-16T03:50:58.570630+0000","last_active":"2024-09-16T04:21:16.872785+0000","last_peered":"2024-09-16T04:21:16.872785+0000","last_clean":"2024-09-16T04:21:16.872785+0000","last_became_active":"2024-09-16T03:50:58.570205+0000","last_became_peered":"2024-09-16T03:50:58.570205+0000","last_unstale":"2024-09-16T04:21:16.872785+0000","last_undegraded":"2024-09-16T04:21:16.872785+0000","last_fullsized":"2024-09-16T04:21:16.872785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:23.406420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876760+0000","last_change":"2024-09-16T03:50:58.545353+0000","last_active":"2024-09-16T04:21:15.876760+0000","last_peered":"2024-09-16T04:21:15.876760+0000","last_clean":"2024-09-16T04:21:15.876760+0000","last_became_active":"2024-09-16T03:50:58.543727+0000","last_became_peered":"2024-09-16T03:50:58.543727+0000","last_unstale":"2024-09-16T04:21:15.876760+0000","last_undegraded":"2024-09-16T04:21:15.876760+0000","last_fullsized":"2024-09-16T04:21:15.876760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:41.271697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871001+0000","last_change":"2024-09-16T03:50:58.550926+0000","last_active":"2024-09-16T04:21:16.871001+0000","last_peered":"2024-09-16T04:21:16.871001+0000","last_clean":"2024-09-16T04:21:16.871001+0000","last_became_active":"2024-09-16T03:50:58.549440+0000","last_became_peered":"2024-09-16T03:50:58.549440+0000","last_unstale":"2024-09-16T04:21:16.871001+0000","last_undegraded":"2024-09-16T04:21:16.871001+0000","last_fullsized":"2024-09-16T04:21:16.871001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:58:40.013005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878320+0000","last_change":"2024-09-16T03:50:58.483761+0000","last_active":"2024-09-16T04:21:16.878320+0000","last_peered":"2024-09-16T04:21:16.878320+0000","last_clean":"2024-09-16T04:21:16.878320+0000","last_became_active":"2024-09-16T03:50:58.483098+0000","last_became_peered":"2024-09-16T03:50:58.483098+0000","last_unstale":"2024-09-16T04:21:16.878320+0000","last_undegraded":"2024-09-16T04:21:16.878320+0000","last_fullsized":"2024-09-16T04:21:16.878320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:30.723960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872415+0000","last_change":"2024-09-16T03:50:58.524457+0000","last_active":"2024-09-16T04:21:16.872415+0000","last_peered":"2024-09-16T04:21:16.872415+0000","last_clean":"2024-09-16T04:21:16.872415+0000","last_became_active":"2024-09-16T03:50:58.524223+0000","last_became_peered":"2024-09-16T03:50:58.524223+0000","last_unstale":"2024-09-16T04:21:16.872415+0000","last_undegraded":"2024-09-16T04:21:16.872415+0000","last_fullsized":"2024-09-16T04:21:16.872415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:41.294641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.875858+0000","last_change":"2024-09-16T03:50:58.520158+0000","last_active":"2024-09-16T04:21:15.875858+0000","last_peered":"2024-09-16T04:21:15.875858+0000","last_clean":"2024-09-16T04:21:15.875858+0000","last_became_active":"2024-09-16T03:50:58.519543+0000","last_became_peered":"2024-09-16T03:50:58.519543+0000","last_unstale":"2024-09-16T04:21:15.875858+0000","last_undegraded":"2024-09-16T04:21:15.875858+0000","last_fullsized":"2024-09-16T04:21:15.875858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:48.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509823,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68480,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303360,"statfs":{"total":96636764160,"available":96566640640,"internally_reserved":0,"allocated":4923392,"data_stored":2365351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67464,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304376,"statfs":{"total":96636764160,"available":96567681024,"internally_reserved":0,"allocated":4071424,"data_stored":1806610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542526,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68328,"kb_used_data":4840,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303512,"statfs":{"total":96636764160,"available":96566796288,"internally_reserved":0,"allocated":4956160,"data_stored":2572914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79928,"kb_used_data":4144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291912,"statfs":{"total":96636764160,"available":96554917888,"internally_reserved":0,"allocated":4243456,"data_stored":1927659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542525,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71604,"kb_used_data":4324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300236,"statfs":{"total":96636764160,"available":96563441664,"internally_reserved":0,"allocated":4427776,"data_stored":2043991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76132,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295708,"statfs":{"total":96636764160,"available":96558804992,"internally_reserved":0,"allocated":4239360,"data_stored":1927640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80112,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291728,"statfs":{"total":96636764160,"available":96554729472,"internally_reserved":0,"allocated":4431872,"data_stored":2044009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76620,"kb_used_data":4996,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295220,"statfs":{"total":96636764160,"available":96558305280,"internally_reserved":0,"allocated":5115904,"data_stored":2675247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:21:19.080 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:21:19.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.175+0000 7f8771f31640 10 monclient: get_monmap_and_config 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.175+0000 7f8771f31640 10 monclient: build_initial_monmap 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.175+0000 7f8771f31640 10 monclient: monmap: 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:19.176432+0000 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:19.176432+0000 2024-09-16T04:21:19.177 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:19.178 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.176+0000 7f8771f31640 1 Processor -- start 2024-09-16T04:21:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.176+0000 7f8771f31640 1 -- start start 2024-09-16T04:21:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.176+0000 7f8771f31640 10 monclient: init 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f876c0b5470 0x7f876c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-a con 0x7f876c0b5470 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f876c13cbd0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f876c13d020 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:19.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c0ac040 con 0x7f876c13cbd0 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876c0ac2b0 con 0x7f876c13d020 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f876c13d450 con 0x7f876c0b5470 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f8771f31640 10 monclient(hunting): authenticate will time out at 2637.838415s 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f876c0b5470 0x7f876c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f876c13cbd0 0x7f876c143040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42440/0 (socket says 172.21.15.70:42440) 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876affd640 1 -- 172.21.15.70:0/2088253096 learned_addr learned my addr 172.21.15.70:0/2088253096 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): get_auth_request con 0x7f876c0b5470 auth_method 0 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient(hunting): handle_auth_done global_id 26718 payload 274 2024-09-16T04:21:19.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 1 -- 172.21.15.70:0/2088253096 >> v1:172.21.15.163:6789/0 conn(0x7f876c13d020 legacy=0x7f876c143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 1 -- 172.21.15.70:0/2088253096 >> v1:172.21.15.70:6789/0 conn(0x7f876c13cbd0 legacy=0x7f876c143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.177+0000 7f876b7fe640 1 -- 172.21.15.70:0/2088253096 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c0aaef0 con 0x7f876c0b5470 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876b7fe640 1 --2- 172.21.15.70:0/2088253096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c0b5470 0x7f876c0b5850 secure :-1 s=READY pgs=5733 cs=0 l=1 rev1=1 crypto rx=0x7f876c13d5d0 tx=0x7f875c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b025c5fe42977a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 1 -- 172.21.15.70:0/2088253096 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c037070 con 0x7f876c0b5470 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: dump: 2024-09-16T04:21:19.193 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f8771f31640 5 monclient: authenticate success, global_id 26718 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 1 -- 172.21.15.70:0/2088253096 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c032440 con 0x7f876c0b5470 2024-09-16T04:21:19.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f8771f31640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f8771f31640 4 set_mon_vals no callback set 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 1 -- 172.21.15.70:0/2088253096 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c032810 con 0x7f876c0b5470 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.178+0000 7f876a7fc640 10 monclient: dump: 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.195 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 10 monclient: shutdown 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 -- 172.21.15.70:0/2088253096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c0b5470 msgr2=0x7f876c0b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 --2- 172.21.15.70:0/2088253096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c0b5470 0x7f876c0b5850 secure :-1 s=READY pgs=5733 cs=0 l=1 rev1=1 crypto rx=0x7f876c13d5d0 tx=0x7f875c02a210 comp rx=0 tx=0).stop 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 -- 172.21.15.70:0/2088253096 shutdown_connections 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 --2- 172.21.15.70:0/2088253096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c0b5470 0x7f876c0b5850 unknown :-1 s=CLOSED pgs=5733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 -- 172.21.15.70:0/2088253096 >> 172.21.15.70:0/2088253096 conn(0x7f876c0679c0 msgr2=0x7f876c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 -- 172.21.15.70:0/2088253096 shutdown_connections 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 1 -- 172.21.15.70:0/2088253096 wait complete. 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 10 monclient: build_initial_monmap 2024-09-16T04:21:19.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.179+0000 7f8771f31640 10 monclient: monmap: 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 librados: starting msgr at 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 librados: starting objecter 2024-09-16T04:21:19.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 Processor -- start 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 -- start start 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 librados: setting wanted keys 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 librados: calling monclient init 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient: init 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f876c0b5470 0x7f876c146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-b con 0x7f876c0b5470 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-c con 0x7f876c13cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 0x7f876c0aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.180+0000 7f8771f31640 10 monclient(hunting): picked mon.noname-a con 0x7f876c13d020 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:19.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f876c0b5f10 con 0x7f876c13d020 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f876c0b5d90 con 0x7f876c0b5470 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f876c14e8d0 con 0x7f876c13cbd0 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient(hunting): authenticate will time out at 2637.842023s 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 0x7f876c0aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54714/0 (socket says 172.21.15.70:54714) 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 -- 172.21.15.70:0/345830105 learned_addr learned my addr 172.21.15.70:0/345830105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): get_auth_request con 0x7f876c13d020 auth_method 0 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): get_auth_request con 0x7f876c13cbd0 auth_method 0 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876b7fe640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f876c0b5470 0x7f876c146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876b7fe640 10 monclient(hunting): get_auth_request con 0x7f876c0b5470 auth_method 0 2024-09-16T04:21:19.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient(hunting): handle_auth_done global_id 20954 payload 824 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 -- 172.21.15.70:0/345830105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f876c0b5470 msgr2=0x7f876c146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f876c0b5470 0x7f876c146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 -- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 msgr2=0x7f876c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 0x7f876c0aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 monclient: found mon.noname-c 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 -- 172.21.15.70:0/345830105 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876c0afd90 con 0x7f876c13cbd0 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876bfff640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 0x7f876c0aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f876affd640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f876000c4a0 tx=0x7f876000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 client.?.objecter ms_handle_connect 0x7f876c13cbd0 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760017070 con 0x7f876c13cbd0 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: dump: 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.202 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:19.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 5 monclient: authenticate success, global_id 20954 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8760005ce0 con 0x7f876c13cbd0 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760007270 con 0x7f876c13cbd0 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8753fff640 10 monclient: dump: 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient: _renew_subs 2024-09-16T04:21:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8769ffb640 4 set_mon_vals no callback set 2024-09-16T04:21:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876c0b00d0 con 0x7f876c13cbd0 2024-09-16T04:21:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 client.20954.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:19.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient: _renew_subs 2024-09-16T04:21:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f876c0b07e0 con 0x7f876c13cbd0 2024-09-16T04:21:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.181+0000 7f8771f31640 1 librados: init done 2024-09-16T04:21:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.182+0000 7f8771f31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.182+0000 7f8771f31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.182+0000 7f8771f31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.182+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f876c0b5850 con 0x7f876c13cbd0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f876001a070 con 0x7f876c13cbd0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 1 --2- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 0x7f87580700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f876b7fe640 1 --2- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 0x7f87580700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f876b7fe640 10 monclient: get_auth_request con 0x7f875806dc00 auth_method 0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f876008a400 con 0x7f876c13cbd0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 10 client.20954.objecter ms_dispatch 0x7f876c000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 3 client.20954.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 3 client.20954.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 20 client.20954.objecter dump_active .. 0 homeless 2024-09-16T04:21:19.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f8753fff640 10 client.20954.objecter ms_handle_connect 0x7f875806dc00 2024-09-16T04:21:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.183+0000 7f876b7fe640 1 --2- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 0x7f87580700c0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7f875c034000 tx=0x7f875c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.187+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f876c0b00d0 con 0x7f876c13cbd0 2024-09-16T04:21:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.187+0000 7f8753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.187+0000 7f8753fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:19.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.351+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f876c0b03b0 con 0x7f875806dc00 2024-09-16T04:21:19.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.354+0000 7f8753fff640 1 -- 172.21.15.70:0/345830105 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116292 (secure 0 0 0) 0x7f876c0b03b0 con 0x7f875806dc00 2024-09-16T04:21:19.356 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:19.356 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T04:21:19.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 10 librados: watch_flush enter 2024-09-16T04:21:19.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 10 librados: watch_flush exit 2024-09-16T04:21:19.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 20 client.20954.objecter shutdown clearing up homeless session... 2024-09-16T04:21:19.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 10 client.20954.objecter successfully canceled tick 2024-09-16T04:21:19.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 msgr2=0x7f87580700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 0x7f87580700c0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7f875c034000 tx=0x7f875c02e000 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 10 monclient: shutdown 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 msgr2=0x7f876c0ae1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f876000c4a0 tx=0x7f876000c960 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 shutdown_connections 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> v2:172.21.15.70:6804/198622594 conn(0x7f875806dc00 0x7f87580700c0 unknown :-1 s=CLOSED pgs=4026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f876c13d020 0x7f876c0aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f876c13cbd0 0x7f876c0ae1d0 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.356+0000 7f8771f31640 1 --2- 172.21.15.70:0/345830105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f876c0b5470 0x7f876c146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.357+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 >> 172.21.15.70:0/345830105 conn(0x7f876c0679c0 msgr2=0x7f876c0b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.357+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 shutdown_connections 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.357+0000 7f8771f31640 1 -- 172.21.15.70:0/345830105 wait complete. 2024-09-16T04:21:19.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.357+0000 7f8771f31640 1 librados: shutdown 2024-09-16T04:21:19.369 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1694,"stamp":"2024-09-16T04:21:19.172992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":587284,"kb_used_data":34724,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754387436,"statfs":{"total":773094113280,"available":772492734464,"internally_reserved":0,"allocated":35557376,"data_stored":16804679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":563793458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.309552"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878232+0000","last_change":"2024-09-16T03:50:58.527921+0000","last_active":"2024-09-16T04:21:16.878232+0000","last_peered":"2024-09-16T04:21:16.878232+0000","last_clean":"2024-09-16T04:21:16.878232+0000","last_became_active":"2024-09-16T03:50:58.527591+0000","last_became_peered":"2024-09-16T03:50:58.527591+0000","last_unstale":"2024-09-16T04:21:16.878232+0000","last_undegraded":"2024-09-16T04:21:16.878232+0000","last_fullsized":"2024-09-16T04:21:16.878232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.155864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872367+0000","last_change":"2024-09-16T03:50:58.566215+0000","last_active":"2024-09-16T04:21:15.872367+0000","last_peered":"2024-09-16T04:21:15.872367+0000","last_clean":"2024-09-16T04:21:15.872367+0000","last_became_active":"2024-09-16T03:50:58.565169+0000","last_became_peered":"2024-09-16T03:50:58.565169+0000","last_unstale":"2024-09-16T04:21:15.872367+0000","last_undegraded":"2024-09-16T04:21:15.872367+0000","last_fullsized":"2024-09-16T04:21:15.872367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:31.209733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872690+0000","last_change":"2024-09-16T03:50:58.521189+0000","last_active":"2024-09-16T04:21:15.872690+0000","last_peered":"2024-09-16T04:21:15.872690+0000","last_clean":"2024-09-16T04:21:15.872690+0000","last_became_active":"2024-09-16T03:50:58.520920+0000","last_became_peered":"2024-09-16T03:50:58.520920+0000","last_unstale":"2024-09-16T04:21:15.872690+0000","last_undegraded":"2024-09-16T04:21:15.872690+0000","last_fullsized":"2024-09-16T04:21:15.872690+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:15.187842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878466+0000","last_change":"2024-09-16T03:50:58.529897+0000","last_active":"2024-09-16T04:21:16.878466+0000","last_peered":"2024-09-16T04:21:16.878466+0000","last_clean":"2024-09-16T04:21:16.878466+0000","last_became_active":"2024-09-16T03:50:58.529616+0000","last_became_peered":"2024-09-16T03:50:58.529616+0000","last_unstale":"2024-09-16T04:21:16.878466+0000","last_undegraded":"2024-09-16T04:21:16.878466+0000","last_fullsized":"2024-09-16T04:21:16.878466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:40.902750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870995+0000","last_change":"2024-09-16T03:50:58.547657+0000","last_active":"2024-09-16T04:21:16.870995+0000","last_peered":"2024-09-16T04:21:16.870995+0000","last_clean":"2024-09-16T04:21:16.870995+0000","last_became_active":"2024-09-16T03:50:58.547418+0000","last_became_peered":"2024-09-16T03:50:58.547418+0000","last_unstale":"2024-09-16T04:21:16.870995+0000","last_undegraded":"2024-09-16T04:21:16.870995+0000","last_fullsized":"2024-09-16T04:21:16.870995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:45.151909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871082+0000","last_change":"2024-09-16T03:50:58.563918+0000","last_active":"2024-09-16T04:21:16.871082+0000","last_peered":"2024-09-16T04:21:16.871082+0000","last_clean":"2024-09-16T04:21:16.871082+0000","last_became_active":"2024-09-16T03:50:58.563681+0000","last_became_peered":"2024-09-16T03:50:58.563681+0000","last_unstale":"2024-09-16T04:21:16.871082+0000","last_undegraded":"2024-09-16T04:21:16.871082+0000","last_fullsized":"2024-09-16T04:21:16.871082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:01.363054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871296+0000","last_change":"2024-09-16T03:50:58.563162+0000","last_active":"2024-09-16T04:21:16.871296+0000","last_peered":"2024-09-16T04:21:16.871296+0000","last_clean":"2024-09-16T04:21:16.871296+0000","last_became_active":"2024-09-16T03:50:58.562917+0000","last_became_peered":"2024-09-16T03:50:58.562917+0000","last_unstale":"2024-09-16T04:21:16.871296+0000","last_undegraded":"2024-09-16T04:21:16.871296+0000","last_fullsized":"2024-09-16T04:21:16.871296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:20.514696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871351+0000","last_change":"2024-09-16T03:50:58.552412+0000","last_active":"2024-09-16T04:21:16.871351+0000","last_peered":"2024-09-16T04:21:16.871351+0000","last_clean":"2024-09-16T04:21:16.871351+0000","last_became_active":"2024-09-16T03:50:58.552137+0000","last_became_peered":"2024-09-16T03:50:58.552137+0000","last_unstale":"2024-09-16T04:21:16.871351+0000","last_undegraded":"2024-09-16T04:21:16.871351+0000","last_fullsized":"2024-09-16T04:21:16.871351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:13.498957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871506+0000","last_change":"2024-09-16T03:50:58.546028+0000","last_active":"2024-09-16T04:21:16.871506+0000","last_peered":"2024-09-16T04:21:16.871506+0000","last_clean":"2024-09-16T04:21:16.871506+0000","last_became_active":"2024-09-16T03:50:58.545776+0000","last_became_peered":"2024-09-16T03:50:58.545776+0000","last_unstale":"2024-09-16T04:21:16.871506+0000","last_undegraded":"2024-09-16T04:21:16.871506+0000","last_fullsized":"2024-09-16T04:21:16.871506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:03.340804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1442,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879550+0000","last_change":"2024-09-16T03:50:58.540624+0000","last_active":"2024-09-16T04:21:15.879550+0000","last_peered":"2024-09-16T04:21:15.879550+0000","last_clean":"2024-09-16T04:21:15.879550+0000","last_became_active":"2024-09-16T03:50:58.540379+0000","last_became_peered":"2024-09-16T03:50:58.540379+0000","last_unstale":"2024-09-16T04:21:15.879550+0000","last_undegraded":"2024-09-16T04:21:15.879550+0000","last_fullsized":"2024-09-16T04:21:15.879550+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:12.307201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879223+0000","last_change":"2024-09-16T03:50:58.480959+0000","last_active":"2024-09-16T04:21:16.879223+0000","last_peered":"2024-09-16T04:21:16.879223+0000","last_clean":"2024-09-16T04:21:16.879223+0000","last_became_active":"2024-09-16T03:50:58.480684+0000","last_became_peered":"2024-09-16T03:50:58.480684+0000","last_unstale":"2024-09-16T04:21:16.879223+0000","last_undegraded":"2024-09-16T04:21:16.879223+0000","last_fullsized":"2024-09-16T04:21:16.879223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:38.866093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1460,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872748+0000","last_change":"2024-09-16T03:50:58.508324+0000","last_active":"2024-09-16T04:21:16.872748+0000","last_peered":"2024-09-16T04:21:16.872748+0000","last_clean":"2024-09-16T04:21:16.872748+0000","last_became_active":"2024-09-16T03:50:58.507923+0000","last_became_peered":"2024-09-16T03:50:58.507923+0000","last_unstale":"2024-09-16T04:21:16.872748+0000","last_undegraded":"2024-09-16T04:21:16.872748+0000","last_fullsized":"2024-09-16T04:21:16.872748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:50.490187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871014+0000","last_change":"2024-09-16T03:50:58.539556+0000","last_active":"2024-09-16T04:21:16.871014+0000","last_peered":"2024-09-16T04:21:16.871014+0000","last_clean":"2024-09-16T04:21:16.871014+0000","last_became_active":"2024-09-16T03:50:58.539098+0000","last_became_peered":"2024-09-16T03:50:58.539098+0000","last_unstale":"2024-09-16T04:21:16.871014+0000","last_undegraded":"2024-09-16T04:21:16.871014+0000","last_fullsized":"2024-09-16T04:21:16.871014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:34.294689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":1616,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871130+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T04:21:16.871130+0000","last_peered":"2024-09-16T04:21:16.871130+0000","last_clean":"2024-09-16T04:21:16.871130+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T04:21:16.871130+0000","last_undegraded":"2024-09-16T04:21:16.871130+0000","last_fullsized":"2024-09-16T04:21:16.871130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":1457,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878405+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T04:21:16.878405+0000","last_peered":"2024-09-16T04:21:16.878405+0000","last_clean":"2024-09-16T04:21:16.878405+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T04:21:16.878405+0000","last_undegraded":"2024-09-16T04:21:16.878405+0000","last_fullsized":"2024-09-16T04:21:16.878405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1439,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.879231+0000","last_change":"2024-09-16T03:50:58.527360+0000","last_active":"2024-09-16T04:21:16.879231+0000","last_peered":"2024-09-16T04:21:16.879231+0000","last_clean":"2024-09-16T04:21:16.879231+0000","last_became_active":"2024-09-16T03:50:58.527023+0000","last_became_peered":"2024-09-16T03:50:58.527023+0000","last_unstale":"2024-09-16T04:21:16.879231+0000","last_undegraded":"2024-09-16T04:21:16.879231+0000","last_fullsized":"2024-09-16T04:21:16.879231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:42.928632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871059+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T04:21:16.871059+0000","last_peered":"2024-09-16T04:21:16.871059+0000","last_clean":"2024-09-16T04:21:16.871059+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T04:21:16.871059+0000","last_undegraded":"2024-09-16T04:21:16.871059+0000","last_fullsized":"2024-09-16T04:21:16.871059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871813+0000","last_change":"2024-09-16T03:50:58.528877+0000","last_active":"2024-09-16T04:21:16.871813+0000","last_peered":"2024-09-16T04:21:16.871813+0000","last_clean":"2024-09-16T04:21:16.871813+0000","last_became_active":"2024-09-16T03:50:58.528663+0000","last_became_peered":"2024-09-16T03:50:58.528663+0000","last_unstale":"2024-09-16T04:21:16.871813+0000","last_undegraded":"2024-09-16T04:21:16.871813+0000","last_fullsized":"2024-09-16T04:21:16.871813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.620543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1463,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871100+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T04:21:16.871100+0000","last_peered":"2024-09-16T04:21:16.871100+0000","last_clean":"2024-09-16T04:21:16.871100+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T04:21:16.871100+0000","last_undegraded":"2024-09-16T04:21:16.871100+0000","last_fullsized":"2024-09-16T04:21:16.871100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865432+0000","last_change":"2024-09-16T03:50:58.548783+0000","last_active":"2024-09-16T04:21:14.865432+0000","last_peered":"2024-09-16T04:21:14.865432+0000","last_clean":"2024-09-16T04:21:14.865432+0000","last_became_active":"2024-09-16T03:50:58.545532+0000","last_became_peered":"2024-09-16T03:50:58.545532+0000","last_unstale":"2024-09-16T04:21:14.865432+0000","last_undegraded":"2024-09-16T04:21:14.865432+0000","last_fullsized":"2024-09-16T04:21:14.865432+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:46.492270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":1458,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878077+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T04:21:16.878077+0000","last_peered":"2024-09-16T04:21:16.878077+0000","last_clean":"2024-09-16T04:21:16.878077+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T04:21:16.878077+0000","last_undegraded":"2024-09-16T04:21:16.878077+0000","last_fullsized":"2024-09-16T04:21:16.878077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1463,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865416+0000","last_change":"2024-09-16T03:50:58.560987+0000","last_active":"2024-09-16T04:21:14.865416+0000","last_peered":"2024-09-16T04:21:14.865416+0000","last_clean":"2024-09-16T04:21:14.865416+0000","last_became_active":"2024-09-16T03:50:58.555850+0000","last_became_peered":"2024-09-16T03:50:58.555850+0000","last_unstale":"2024-09-16T04:21:14.865416+0000","last_undegraded":"2024-09-16T04:21:14.865416+0000","last_fullsized":"2024-09-16T04:21:14.865416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:43.936131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1459,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.879705+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T04:21:15.879705+0000","last_peered":"2024-09-16T04:21:15.879705+0000","last_clean":"2024-09-16T04:21:15.879705+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T04:21:15.879705+0000","last_undegraded":"2024-09-16T04:21:15.879705+0000","last_fullsized":"2024-09-16T04:21:15.879705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:43:20.389723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881223+0000","last_change":"2024-09-16T03:50:58.561772+0000","last_active":"2024-09-16T04:21:17.881223+0000","last_peered":"2024-09-16T04:21:17.881223+0000","last_clean":"2024-09-16T04:21:17.881223+0000","last_became_active":"2024-09-16T03:50:58.561522+0000","last_became_peered":"2024-09-16T03:50:58.561522+0000","last_unstale":"2024-09-16T04:21:17.881223+0000","last_undegraded":"2024-09-16T04:21:17.881223+0000","last_fullsized":"2024-09-16T04:21:17.881223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:46.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1464,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871185+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T04:21:16.871185+0000","last_peered":"2024-09-16T04:21:16.871185+0000","last_clean":"2024-09-16T04:21:16.871185+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T04:21:16.871185+0000","last_undegraded":"2024-09-16T04:21:16.871185+0000","last_fullsized":"2024-09-16T04:21:16.871185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"24'7","reported_seq":1474,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872634+0000","last_change":"2024-09-16T03:50:58.476506+0000","last_active":"2024-09-16T04:21:16.872634+0000","last_peered":"2024-09-16T04:21:16.872634+0000","last_clean":"2024-09-16T04:21:16.872634+0000","last_became_active":"2024-09-16T03:50:58.476200+0000","last_became_peered":"2024-09-16T03:50:58.476200+0000","last_unstale":"2024-09-16T04:21:16.872634+0000","last_undegraded":"2024-09-16T04:21:16.872634+0000","last_fullsized":"2024-09-16T04:21:16.872634+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:05.408395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1458,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876027+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T04:21:15.876027+0000","last_peered":"2024-09-16T04:21:15.876027+0000","last_clean":"2024-09-16T04:21:15.876027+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T04:21:15.876027+0000","last_undegraded":"2024-09-16T04:21:15.876027+0000","last_fullsized":"2024-09-16T04:21:15.876027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:00.954047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1463,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872030+0000","last_change":"2024-09-16T03:50:58.522695+0000","last_active":"2024-09-16T04:21:15.872030+0000","last_peered":"2024-09-16T04:21:15.872030+0000","last_clean":"2024-09-16T04:21:15.872030+0000","last_became_active":"2024-09-16T03:50:58.522014+0000","last_became_peered":"2024-09-16T03:50:58.522014+0000","last_unstale":"2024-09-16T04:21:15.872030+0000","last_undegraded":"2024-09-16T04:21:15.872030+0000","last_fullsized":"2024-09-16T04:21:15.872030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:38.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1473,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.882958+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T04:21:17.882958+0000","last_peered":"2024-09-16T04:21:17.882958+0000","last_clean":"2024-09-16T04:21:17.882958+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T04:21:17.882958+0000","last_undegraded":"2024-09-16T04:21:17.882958+0000","last_fullsized":"2024-09-16T04:21:17.882958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.877443+0000","last_change":"2024-09-16T03:50:58.520798+0000","last_active":"2024-09-16T04:21:15.877443+0000","last_peered":"2024-09-16T04:21:15.877443+0000","last_clean":"2024-09-16T04:21:15.877443+0000","last_became_active":"2024-09-16T03:50:58.519191+0000","last_became_peered":"2024-09-16T03:50:58.519191+0000","last_unstale":"2024-09-16T04:21:15.877443+0000","last_undegraded":"2024-09-16T04:21:15.877443+0000","last_fullsized":"2024-09-16T04:21:15.877443+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:15.389894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1460,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.872620+0000","last_change":"2024-09-16T03:50:58.547695+0000","last_active":"2024-09-16T04:21:15.872620+0000","last_peered":"2024-09-16T04:21:15.872620+0000","last_clean":"2024-09-16T04:21:15.872620+0000","last_became_active":"2024-09-16T03:50:58.546981+0000","last_became_peered":"2024-09-16T03:50:58.546981+0000","last_unstale":"2024-09-16T04:21:15.872620+0000","last_undegraded":"2024-09-16T04:21:15.872620+0000","last_fullsized":"2024-09-16T04:21:15.872620+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:29.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.870965+0000","last_change":"2024-09-16T03:50:58.542578+0000","last_active":"2024-09-16T04:21:16.870965+0000","last_peered":"2024-09-16T04:21:16.870965+0000","last_clean":"2024-09-16T04:21:16.870965+0000","last_became_active":"2024-09-16T03:50:58.541780+0000","last_became_peered":"2024-09-16T03:50:58.541780+0000","last_unstale":"2024-09-16T04:21:16.870965+0000","last_undegraded":"2024-09-16T04:21:16.870965+0000","last_fullsized":"2024-09-16T04:21:16.870965+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:34.596173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881253+0000","last_change":"2024-09-16T03:50:58.542966+0000","last_active":"2024-09-16T04:21:17.881253+0000","last_peered":"2024-09-16T04:21:17.881253+0000","last_clean":"2024-09-16T04:21:17.881253+0000","last_became_active":"2024-09-16T03:50:58.542726+0000","last_became_peered":"2024-09-16T03:50:58.542726+0000","last_unstale":"2024-09-16T04:21:17.881253+0000","last_undegraded":"2024-09-16T04:21:17.881253+0000","last_fullsized":"2024-09-16T04:21:17.881253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:11.421727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1464,"reported_epoch":751,"state":"active+clean","last_fresh":"2024-09-16T04:21:14.865544+0000","last_change":"2024-09-16T03:50:58.558551+0000","last_active":"2024-09-16T04:21:14.865544+0000","last_peered":"2024-09-16T04:21:14.865544+0000","last_clean":"2024-09-16T04:21:14.865544+0000","last_became_active":"2024-09-16T03:50:58.557890+0000","last_became_peered":"2024-09-16T03:50:58.557890+0000","last_unstale":"2024-09-16T04:21:14.865544+0000","last_undegraded":"2024-09-16T04:21:14.865544+0000","last_fullsized":"2024-09-16T04:21:14.865544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:13.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.882902+0000","last_change":"2024-09-16T03:50:58.545310+0000","last_active":"2024-09-16T04:21:17.882902+0000","last_peered":"2024-09-16T04:21:17.882902+0000","last_clean":"2024-09-16T04:21:17.882902+0000","last_became_active":"2024-09-16T03:50:58.544536+0000","last_became_peered":"2024-09-16T03:50:58.544536+0000","last_unstale":"2024-09-16T04:21:17.882902+0000","last_undegraded":"2024-09-16T04:21:17.882902+0000","last_fullsized":"2024-09-16T04:21:17.882902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:34:15.701626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872785+0000","last_change":"2024-09-16T03:50:58.570630+0000","last_active":"2024-09-16T04:21:16.872785+0000","last_peered":"2024-09-16T04:21:16.872785+0000","last_clean":"2024-09-16T04:21:16.872785+0000","last_became_active":"2024-09-16T03:50:58.570205+0000","last_became_peered":"2024-09-16T03:50:58.570205+0000","last_unstale":"2024-09-16T04:21:16.872785+0000","last_undegraded":"2024-09-16T04:21:16.872785+0000","last_fullsized":"2024-09-16T04:21:16.872785+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:23.406420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.876760+0000","last_change":"2024-09-16T03:50:58.545353+0000","last_active":"2024-09-16T04:21:15.876760+0000","last_peered":"2024-09-16T04:21:15.876760+0000","last_clean":"2024-09-16T04:21:15.876760+0000","last_became_active":"2024-09-16T03:50:58.543727+0000","last_became_peered":"2024-09-16T03:50:58.543727+0000","last_unstale":"2024-09-16T04:21:15.876760+0000","last_undegraded":"2024-09-16T04:21:15.876760+0000","last_fullsized":"2024-09-16T04:21:15.876760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:41.271697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.871001+0000","last_change":"2024-09-16T03:50:58.550926+0000","last_active":"2024-09-16T04:21:16.871001+0000","last_peered":"2024-09-16T04:21:16.871001+0000","last_clean":"2024-09-16T04:21:16.871001+0000","last_became_active":"2024-09-16T03:50:58.549440+0000","last_became_peered":"2024-09-16T03:50:58.549440+0000","last_unstale":"2024-09-16T04:21:16.871001+0000","last_undegraded":"2024-09-16T04:21:16.871001+0000","last_fullsized":"2024-09-16T04:21:16.871001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:58:40.013005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1438,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.878320+0000","last_change":"2024-09-16T03:50:58.483761+0000","last_active":"2024-09-16T04:21:16.878320+0000","last_peered":"2024-09-16T04:21:16.878320+0000","last_clean":"2024-09-16T04:21:16.878320+0000","last_became_active":"2024-09-16T03:50:58.483098+0000","last_became_peered":"2024-09-16T03:50:58.483098+0000","last_unstale":"2024-09-16T04:21:16.878320+0000","last_undegraded":"2024-09-16T04:21:16.878320+0000","last_fullsized":"2024-09-16T04:21:16.878320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:30.723960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1459,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-16T04:21:16.872415+0000","last_change":"2024-09-16T03:50:58.524457+0000","last_active":"2024-09-16T04:21:16.872415+0000","last_peered":"2024-09-16T04:21:16.872415+0000","last_clean":"2024-09-16T04:21:16.872415+0000","last_became_active":"2024-09-16T03:50:58.524223+0000","last_became_peered":"2024-09-16T03:50:58.524223+0000","last_unstale":"2024-09-16T04:21:16.872415+0000","last_undegraded":"2024-09-16T04:21:16.872415+0000","last_fullsized":"2024-09-16T04:21:16.872415+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:41.294641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1441,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-09-16T04:21:15.875858+0000","last_change":"2024-09-16T03:50:58.520158+0000","last_active":"2024-09-16T04:21:15.875858+0000","last_peered":"2024-09-16T04:21:15.875858+0000","last_clean":"2024-09-16T04:21:15.875858+0000","last_became_active":"2024-09-16T03:50:58.519543+0000","last_became_peered":"2024-09-16T03:50:58.519543+0000","last_unstale":"2024-09-16T04:21:15.875858+0000","last_undegraded":"2024-09-16T04:21:15.875858+0000","last_fullsized":"2024-09-16T04:21:15.875858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:48.665464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67096,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304744,"statfs":{"total":96636764160,"available":96568057856,"internally_reserved":0,"allocated":4071424,"data_stored":1806609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67464,"kb_used_data":3976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304376,"statfs":{"total":96636764160,"available":96567681024,"internally_reserved":0,"allocated":4071424,"data_stored":1806610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542526,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68328,"kb_used_data":4840,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303512,"statfs":{"total":96636764160,"available":96566796288,"internally_reserved":0,"allocated":4956160,"data_stored":2572914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542526,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79928,"kb_used_data":4144,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291912,"statfs":{"total":96636764160,"available":96554917888,"internally_reserved":0,"allocated":4243456,"data_stored":1927659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542525,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71604,"kb_used_data":4324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300236,"statfs":{"total":96636764160,"available":96563441664,"internally_reserved":0,"allocated":4427776,"data_stored":2043991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542527,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76132,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295708,"statfs":{"total":96636764160,"available":96558804992,"internally_reserved":0,"allocated":4239360,"data_stored":1927640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542525,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80112,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291728,"statfs":{"total":96636764160,"available":96554729472,"internally_reserved":0,"allocated":4431872,"data_stored":2044009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575230,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76620,"kb_used_data":4996,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295220,"statfs":{"total":96636764160,"available":96558305280,"internally_reserved":0,"allocated":5115904,"data_stored":2675247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:21:19.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:21:19.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.466+0000 7f3c03a7b640 10 monclient: get_monmap_and_config 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.466+0000 7f3c03a7b640 10 monclient: build_initial_monmap 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.466+0000 7f3c03a7b640 10 monclient: monmap: 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:19.467252+0000 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:19.467252+0000 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:19.469 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:19.469 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.467+0000 7f3c03a7b640 1 Processor -- start 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.467+0000 7f3c03a7b640 1 -- start start 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.467+0000 7f3c03a7b640 10 monclient: init 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3bfc0acb60 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bfc0af550 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3bfc13e3a0 0x7f3bfc14c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc13e3a0 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bfc0aeaf0 con 0x7f3bfc0acb60 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bfc0aedd0 con 0x7f3bfc0af550 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3bfc0af0b0 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:19.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c03a7b640 10 monclient(hunting): authenticate will time out at 2638.129163s 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c01817640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3bfc0acb60 0x7f3bfc141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42444/0 (socket says 172.21.15.70:42444) 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c01817640 1 -- 172.21.15.70:0/312706140 learned_addr learned my addr 172.21.15.70:0/312706140 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 1 --2- 172.21.15.70:0/312706140 >> v2:172.21.15.70:3300/0 conn(0x7f3bfc13e3a0 0x7f3bfc14c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): get_auth_request con 0x7f3bfc13e3a0 auth_method 0 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 1 -- 172.21.15.70:0/312706140 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 753826574 0 0) 0x7f3bfc0aeaf0 con 0x7f3bfc0acb60 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 10 monclient(hunting): _negotiate 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 10 monclient(hunting): my global_id is 26730 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c00815640 1 -- 172.21.15.70:0/312706140 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3be8003520 con 0x7f3bfc0acb60 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient(hunting): handle_auth_done global_id 26727 payload 274 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 1 -- 172.21.15.70:0/312706140 >> v1:172.21.15.163:6789/0 conn(0x7f3bfc0af550 legacy=0x7f3bfc13dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 1 -- 172.21.15.70:0/312706140 >> v1:172.21.15.70:6789/0 conn(0x7f3bfc0acb60 legacy=0x7f3bfc141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient: _finish_hunting 0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 1 monclient: found mon.noname-a 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.468+0000 7f3c02018640 1 -- 172.21.15.70:0/312706140 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc140e50 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c02018640 1 --2- 172.21.15.70:0/312706140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc13e3a0 0x7f3bfc14c560 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14bbf0f82adf2a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 1 -- 172.21.15.70:0/312706140 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4007740 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: dump: 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: _finish_auth 0 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: _check_auth_tickets 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 5 monclient: authenticate success, global_id 26727 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 1 -- 172.21.15.70:0/312706140 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf40078e0 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 1 -- 172.21.15.70:0/312706140 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4007cb0 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 4 set_mon_vals no callback set 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c00815640 10 monclient: dump: 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 10 monclient: shutdown 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 1 -- 172.21.15.70:0/312706140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc13e3a0 msgr2=0x7f3bfc14c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.469+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/312706140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc13e3a0 0x7f3bfc14c560 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009d90 tx=0x7f3bf402a730 comp rx=0 tx=0).stop 2024-09-16T04:21:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 -- 172.21.15.70:0/312706140 shutdown_connections 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/312706140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc13e3a0 0x7f3bfc14c560 unknown :-1 s=CLOSED pgs=5735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 -- 172.21.15.70:0/312706140 >> 172.21.15.70:0/312706140 conn(0x7f3bfc0b1830 msgr2=0x7f3bfc0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 -- 172.21.15.70:0/312706140 shutdown_connections 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 -- 172.21.15.70:0/312706140 wait complete. 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 10 monclient: build_initial_monmap 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 10 monclient: monmap: 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 librados: starting msgr at 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.470+0000 7f3c03a7b640 1 librados: starting objecter 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 Processor -- start 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 -- start start 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 librados: setting wanted keys 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 librados: calling monclient init 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient: init 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bfc0acb60 0x7f3bfc144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc0acb60 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc0af550 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 0x7f3bfc06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc13e3a0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc0ad0c0 con 0x7f3bfc0af550 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc0acf40 con 0x7f3bfc0acb60 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3bfc0ad240 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c03a7b640 10 monclient(hunting): authenticate will time out at 2638.132761s 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33292/0 (socket says 172.21.15.70:33292) 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 0x7f3bfc06ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 1 -- 172.21.15.70:0/190246993 learned_addr learned my addr 172.21.15.70:0/190246993 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): get_auth_request con 0x7f3bfc0af550 auth_method 0 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): get_auth_request con 0x7f3bfc13e3a0 auth_method 0 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01016640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c01817640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bfc0acb60 0x7f3bfc144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.471+0000 7f3c02018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): get_auth_request con 0x7f3bfc0acb60 auth_method 0 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 10 monclient(hunting): handle_auth_done global_id 20957 payload 824 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 -- 172.21.15.70:0/190246993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bfc0acb60 msgr2=0x7f3bfc144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bfc0acb60 0x7f3bfc144380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 -- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 msgr2=0x7f3bfc0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 10 monclient: _finish_hunting 0 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 monclient: found mon.noname-c 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 -- 172.21.15.70:0/190246993 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc070340 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c01016640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c02018640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 0x7f3bfc06ee50 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4031000 tx=0x7f3bf4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 client.?.objecter ms_handle_connect 0x7f3bfc13e3a0 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf403f070 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: dump: 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf4031c40 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 5 monclient: authenticate success, global_id 20957 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 10 monclient: _renew_subs 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc070620 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4009450 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf27fc640 10 monclient: dump: 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3bf3fff640 4 set_mon_vals no callback set 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 10 client.20957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 10 monclient: _renew_subs 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc06f390 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.472+0000 7f3c03a7b640 1 librados: init done 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.473+0000 7f3c03a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.473+0000 7f3c03a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.473+0000 7f3c03a7b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.473+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc141370 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f3bf4044460 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 1 --2- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 0x7f3be00700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f3bf40ac760 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3c01817640 1 --2- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 0x7f3be00700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 10 client.20957.objecter ms_dispatch 0x7f3bfc000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 3 client.20957.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 3 client.20957.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3c01817640 10 monclient: get_auth_request con 0x7f3be006dc30 auth_method 0 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 20 client.20957.objecter dump_active .. 0 homeless 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3bf27fc640 10 client.20957.objecter ms_handle_connect 0x7f3be006dc30 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.474+0000 7f3c01817640 1 --2- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 0x7f3be00700f0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005880 tx=0x7f3bf8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.478+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3bf407aad0 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.478+0000 7f3bf27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.478+0000 7f3bf27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:19.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.644+0000 7f3c03a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:21:19.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.644+0000 7f3c03a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:21:19.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.644+0000 7f3c03a7b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:19.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.644+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bfc141860 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.645+0000 7f3bf27fc640 1 -- 172.21.15.70:0/190246993 <== mon.2 v2:172.21.15.70:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v754) ==== 74+0+15939 (secure 0 0 0) 0x7f3bf407edd0 con 0x7f3bfc13e3a0 2024-09-16T04:21:19.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.645+0000 7f3bf27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:21:19.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.645+0000 7f3bf27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:21:19.647 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:19.647 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":754,"fsid":"33825cf0-3459-43e2-ae47-46b331d213a2","created":"2024-09-16T03:49:34.524106+0000","modified":"2024-09-16T04:21:17.872876+0000","last_up_change":"2024-09-16T03:49:46.819202+0000","last_in_change":"2024-09-16T03:49:36.644668+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":105,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:49:46.663530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:49:52.440963+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-16T03:50:57.335754+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0e4116b6-869c-4008-a494-02c6bf40f9a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1243681362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6801","nonce":1243681362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":1243681362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1243681362}]},"public_addr":"172.21.15.70:6800/1243681362","cluster_addr":"172.21.15.70:6801/1243681362","heartbeat_back_addr":"172.21.15.70:6803/1243681362","heartbeat_front_addr":"172.21.15.70:6802/1243681362","state":["exists","up"]},{"osd":1,"uuid":"b9adee7c-8804-49bc-894f-8f7461210815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":1499120614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":1499120614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":1499120614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":1499120614}]},"public_addr":"172.21.15.70:6813/1499120614","cluster_addr":"172.21.15.70:6814/1499120614","heartbeat_back_addr":"172.21.15.70:6816/1499120614","heartbeat_front_addr":"172.21.15.70:6815/1499120614","state":["exists","up"]},{"osd":2,"uuid":"9d85899b-73b5-4c63-9962-f74c96a2d208","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":2584301924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2584301924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2584301924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6811","nonce":2584301924}]},"public_addr":"172.21.15.70:6809/2584301924","cluster_addr":"172.21.15.70:6810/2584301924","heartbeat_back_addr":"172.21.15.70:6812/2584301924","heartbeat_front_addr":"172.21.15.70:6811/2584301924","state":["exists","up"]},{"osd":3,"uuid":"d31b8445-ae1e-4bfb-9588-5af8c9daf748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3986044637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3986044637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3986044637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3986044637}]},"public_addr":"172.21.15.70:6805/3986044637","cluster_addr":"172.21.15.70:6806/3986044637","heartbeat_back_addr":"172.21.15.70:6808/3986044637","heartbeat_front_addr":"172.21.15.70:6807/3986044637","state":["exists","up"]},{"osd":4,"uuid":"4a3eca33-e8df-4b98-9fba-9dd4a9298cc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":1904387775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6805","nonce":1904387775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6807","nonce":1904387775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":1904387775}]},"public_addr":"172.21.15.163:6804/1904387775","cluster_addr":"172.21.15.163:6805/1904387775","heartbeat_back_addr":"172.21.15.163:6807/1904387775","heartbeat_front_addr":"172.21.15.163:6806/1904387775","state":["exists","up"]},{"osd":5,"uuid":"3da90f35-297c-4a03-921c-428685f5e5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":1625318059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6809","nonce":1625318059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6811","nonce":1625318059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":1625318059}]},"public_addr":"172.21.15.163:6808/1625318059","cluster_addr":"172.21.15.163:6809/1625318059","heartbeat_back_addr":"172.21.15.163:6811/1625318059","heartbeat_front_addr":"172.21.15.163:6810/1625318059","state":["exists","up"]},{"osd":6,"uuid":"0008e277-c3bd-45c8-a0f6-82494d859a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":2999981402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6813","nonce":2999981402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6815","nonce":2999981402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":2999981402}]},"public_addr":"172.21.15.163:6812/2999981402","cluster_addr":"172.21.15.163:6813/2999981402","heartbeat_back_addr":"172.21.15.163:6815/2999981402","heartbeat_front_addr":"172.21.15.163:6814/2999981402","state":["exists","up"]},{"osd":7,"uuid":"e67e5a12-f19c-485d-8435-b280ab0335db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":751,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3710554613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6801","nonce":3710554613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6803","nonce":3710554613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3710554613}]},"public_addr":"172.21.15.163:6800/3710554613","cluster_addr":"172.21.15.163:6801/3710554613","heartbeat_back_addr":"172.21.15.163:6803/3710554613","heartbeat_front_addr":"172.21.15.163:6802/3710554613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.496124+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.974891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.248106+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.668425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.919888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:44.102132+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.952245+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:49:43.672498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":3}]},{"pool":15,"snaps":[{"begin":2,"length":5}]},{"pool":35,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":133,"snaps":[{"begin":1,"length":4}]},{"pool":140,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":3,"length":4}]},{"pool":157,"snaps":[{"begin":2,"length":4}]},{"pool":168,"snaps":[{"begin":2,"length":1}]},{"pool":173,"snaps":[{"begin":1,"length":4}]},{"pool":186,"snaps":[{"begin":4,"length":4}]},{"pool":195,"snaps":[{"begin":7,"length":1}]},{"pool":217,"snaps":[{"begin":8,"length":1}]},{"pool":228,"snaps":[{"begin":10,"length":1}]},{"pool":233,"snaps":[{"begin":11,"length":2}]},{"pool":244,"snaps":[{"begin":14,"length":1}]},{"pool":293,"snaps":[{"begin":1,"length":1}]},{"pool":295,"snaps":[{"begin":1,"length":1}]},{"pool":296,"snaps":[{"begin":1,"length":2}]},{"pool":297,"snaps":[{"begin":3,"length":1}]},{"pool":299,"snaps":[{"begin":2,"length":2}]},{"pool":300,"snaps":[{"begin":2,"length":3}]},{"pool":302,"snaps":[{"begin":2,"length":2}]},{"pool":315,"snaps":[{"begin":2,"length":1}]},{"pool":316,"snaps":[{"begin":3,"length":1}]},{"pool":319,"snaps":[{"begin":4,"length":1}]},{"pool":323,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 10 librados: watch_flush enter 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 10 librados: watch_flush exit 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 20 client.20957.objecter shutdown clearing up homeless session... 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 10 client.20957.objecter successfully canceled tick 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 msgr2=0x7f3be00700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 0x7f3be00700f0 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8005880 tx=0x7f3bf8009040 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 10 monclient: shutdown 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 msgr2=0x7f3bfc06ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.647+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 0x7f3bfc06ee50 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4031000 tx=0x7f3bf4004a40 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 shutdown_connections 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> v2:172.21.15.70:6804/198622594 conn(0x7f3be006dc30 0x7f3be00700f0 unknown :-1 s=CLOSED pgs=4027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3bfc13e3a0 0x7f3bfc06ee50 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bfc0af550 0x7f3bfc0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 --2- 172.21.15.70:0/190246993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3bfc0acb60 0x7f3bfc144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 >> 172.21.15.70:0/190246993 conn(0x7f3bfc0b1830 msgr2=0x7f3bfc068110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 shutdown_connections 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 -- 172.21.15.70:0/190246993 wait complete. 2024-09-16T04:21:19.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:19.648+0000 7f3c03a7b640 1 librados: shutdown 2024-09-16T04:21:20.661 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:21:20.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.773+0000 7f2eed49b640 10 monclient: get_monmap_and_config 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.773+0000 7f2eed49b640 10 monclient: build_initial_monmap 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.774+0000 7f2eed49b640 10 monclient: monmap: 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:20.774486+0000 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:20.774486+0000 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.775 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:20.776 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:20.776 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:20.776 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:20.776 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:20.776 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.774+0000 7f2eed49b640 1 Processor -- start 2024-09-16T04:21:20.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.774+0000 7f2eed49b640 1 -- start start 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.774+0000 7f2eed49b640 10 monclient: init 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 0x7f2ee8140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee806c2c0 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:20.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 0x7f2ee8146910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee8140a70 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ee8146e50 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee80af050 con 0x7f2ee8146e50 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2ee806c6a0 con 0x7f2ee806c2c0 2024-09-16T04:21:20.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee80af6e0 con 0x7f2ee8140a70 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2eed49b640 10 monclient(hunting): authenticate will time out at 2639.436493s 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 0x7f2ee8146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 0x7f2ee8140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2ee8146e50 0x7f2ee8143060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42448/0 (socket says 172.21.15.70:42448) 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee77fe640 1 -- 172.21.15.70:0/3162411698 learned_addr learned my addr 172.21.15.70:0/3162411698 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f2ee806c2c0 auth_method 0 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): get_auth_request con 0x7f2ee8140a70 auth_method 0 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_done global_id 20960 payload 274 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 -- 172.21.15.70:0/3162411698 >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 msgr2=0x7f2ee8140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:20.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 --2- 172.21.15.70:0/3162411698 >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 0x7f2ee8140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 -- 172.21.15.70:0/3162411698 >> v1:172.21.15.70:6789/0 conn(0x7f2ee8146e50 legacy=0x7f2ee8143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 monclient: found mon.noname-b 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee67fc640 1 -- 172.21.15.70:0/3162411698 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee80af2c0 con 0x7f2ee8140a70 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.775+0000 7f2ee6ffd640 1 --2- 172.21.15.70:0/3162411698 >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 0x7f2ee8140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee67fc640 1 --2- 172.21.15.70:0/3162411698 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 0x7f2ee8146910 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e6c18c69bef6c92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 1 -- 172.21.15.70:0/3162411698 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc037070 con 0x7f2ee8140a70 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:20.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: dump: 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.795 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:20.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2eed49b640 5 monclient: authenticate success, global_id 20960 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 1 -- 172.21.15.70:0/3162411698 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc02adb0 con 0x7f2ee8140a70 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2eed49b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:20.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2eed49b640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 1 -- 172.21.15.70:0/3162411698 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc032660 con 0x7f2ee8140a70 2024-09-16T04:21:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2eed49b640 4 set_mon_vals no callback set 2024-09-16T04:21:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:20.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.776+0000 7f2ee5ffb640 10 monclient: dump: 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.799 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 10 monclient: shutdown 2024-09-16T04:21:20.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 -- 172.21.15.70:0/3162411698 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 msgr2=0x7f2ee8146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 --2- 172.21.15.70:0/3162411698 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 0x7f2ee8146910 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097a0 tx=0x7f2edc02a060 comp rx=0 tx=0).stop 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 -- 172.21.15.70:0/3162411698 shutdown_connections 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 --2- 172.21.15.70:0/3162411698 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8140a70 0x7f2ee8146910 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 --2- 172.21.15.70:0/3162411698 >> v2:172.21.15.70:3300/0 conn(0x7f2ee806c2c0 0x7f2ee8140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 -- 172.21.15.70:0/3162411698 >> 172.21.15.70:0/3162411698 conn(0x7f2ee80685b0 msgr2=0x7f2ee80b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 -- 172.21.15.70:0/3162411698 shutdown_connections 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.777+0000 7f2eed49b640 1 -- 172.21.15.70:0/3162411698 wait complete. 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 10 monclient: build_initial_monmap 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 10 monclient: monmap: 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:20.802 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:20.803 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:20.804 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:20.804 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 librados: starting msgr at 2024-09-16T04:21:20.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 librados: starting objecter 2024-09-16T04:21:20.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 Processor -- start 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 -- start start 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 librados: setting wanted keys 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 1 librados: calling monclient init 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.778+0000 7f2eed49b640 10 monclient: init 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:20.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ee806c2c0 0x7f2ee8072c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee806c2c0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8146e50 0x7f2ee8143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee8146e50 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 0x7f2ee8140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee8142b90 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee80b0d00 con 0x7f2ee8142b90 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee806d3a0 con 0x7f2ee806c2c0 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee8147300 con 0x7f2ee8146e50 2024-09-16T04:21:20.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:20.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2eed49b640 10 monclient(hunting): authenticate will time out at 2639.440317s 2024-09-16T04:21:20.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8146e50 0x7f2ee8143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 0x7f2ee8140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 0x7f2ee8140e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33304/0 (socket says 172.21.15.70:33304) 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 1 -- 172.21.15.70:0/1388460464 learned_addr learned my addr 172.21.15.70:0/1388460464 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): get_auth_request con 0x7f2ee8142b90 auth_method 0 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): get_auth_request con 0x7f2ee8146e50 auth_method 0 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ee806c2c0 0x7f2ee8072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f2ee806c2c0 auth_method 0 2024-09-16T04:21:20.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient(hunting): handle_auth_done global_id 20963 payload 824 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 -- 172.21.15.70:0/1388460464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ee806c2c0 msgr2=0x7f2ee8072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ee806c2c0 0x7f2ee8072c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 -- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 msgr2=0x7f2ee8140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 0x7f2ee8140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 monclient: found mon.noname-c 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 -- 172.21.15.70:0/1388460464 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee81426b0 con 0x7f2ee8146e50 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee77fe640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ee8142b90 0x7f2ee8140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ee67fc640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8146e50 0x7f2ee8143510 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80af860 tx=0x7f2edc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ecf7fe640 10 client.?.objecter ms_handle_connect 0x7f2ee8146e50 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.779+0000 7f2ecf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc041030 con 0x7f2ee8146e50 2024-09-16T04:21:20.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:20.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: dump: 2024-09-16T04:21:20.816 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.816 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.816 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.817 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.818 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:20.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:20.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:20.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc02abb0 con 0x7f2ee8146e50 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 5 monclient: authenticate success, global_id 20963 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 10 monclient: _renew_subs 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee811ef40 con 0x7f2ee8146e50 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:20.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc037040 con 0x7f2ee8146e50 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ecf7fe640 10 monclient: dump: 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.820 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.821 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.821 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.821 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.821 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.821 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 10 client.20963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 10 monclient: _renew_subs 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee81413a0 con 0x7f2ee8146e50 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2eed49b640 1 librados: init done 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.780+0000 7f2ee57fa640 4 set_mon_vals no callback set 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 10 client.20963.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 15 client.20963.objecter _session_command_op_assign -1 1 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 20 client.20963.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:20.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 10 client.20963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:20.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ee8142880 con 0x7f2ee8146e50 2024-09-16T04:21:20.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.781+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f2edc044070 con 0x7f2ee8146e50 2024-09-16T04:21:20.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6804/198622594 conn(0x7f2ed406dc00 0x7f2ed40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee6ffd640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6804/198622594 conn(0x7f2ed406dc00 0x7f2ed40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f2edc0ab670 con 0x7f2ee8146e50 2024-09-16T04:21:20.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 client.20963.objecter ms_dispatch 0x7f2ee8000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:20.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 3 client.20963.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:20.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 3 client.20963.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:20.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee6ffd640 10 monclient: get_auth_request con 0x7f2ed406dc00 auth_method 0 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 client.20963.objecter checking command 1 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6800/1243681362 conn(0x7f2ed4075520 0x7f2ed40779c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter _get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 15 client.20963.objecter _session_command_op_remove -1 1 2024-09-16T04:21:20.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter _get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 15 client.20963.objecter _session_command_op_assign 0 1 2024-09-16T04:21:20.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 4 2024-09-16T04:21:20.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 client.20963.objecter _send_command 1 2024-09-16T04:21:20.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 --> v2:172.21.15.70:6800/1243681362 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2edc001d30 con 0x7f2ed4075520 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 20 client.20963.objecter dump_active .. 0 homeless 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7f2edc0b2050 con 0x7f2ee8146e50 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee77fe640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6800/1243681362 conn(0x7f2ed4075520 0x7f2ed40779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee77fe640 10 monclient: get_auth_request con 0x7f2ed4075520 auth_method 0 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 client.20963.objecter ms_handle_connect 0x7f2ed406dc00 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee6ffd640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6804/198622594 conn(0x7f2ed406dc00 0x7f2ed40700c0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009560 tx=0x7f2ed8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ecf7fe640 10 client.20963.objecter ms_handle_connect 0x7f2ed4075520 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.782+0000 7f2ee77fe640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6800/1243681362 conn(0x7f2ed4075520 0x7f2ed40779c0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.783+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== osd.0 v2:172.21.15.70:6800/1243681362 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2edc001d30 con 0x7f2ed4075520 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.783+0000 7f2ecf7fe640 10 client.20963.objecter ms_dispatch 0x7f2ee8000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.783+0000 7f2ecf7fe640 10 client.20963.objecter _finish_command 1 = system:0 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.783+0000 7f2ecf7fe640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 4 2024-09-16T04:21:20.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.783+0000 7f2ecf7fe640 15 client.20963.objecter _session_command_op_remove 0 1 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 10 client.20963.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 15 client.20963.objecter _session_command_op_assign -1 2 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 20 client.20963.objecter _get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 20 client.20963.objecter _get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 15 client.20963.objecter _session_command_op_remove -1 2 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 20 client.20963.objecter get_session s=0x7f2ed4074e70 osd=0 3 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 15 client.20963.objecter _session_command_op_assign 0 2 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 4 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 10 client.20963.objecter _send_command 2 2024-09-16T04:21:20.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.795+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 --> v2:172.21.15.70:6800/1243681362 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2ee8141690 con 0x7f2ed4075520 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ecf7fe640 1 -- 172.21.15.70:0/1388460464 <== osd.0 v2:172.21.15.70:6800/1243681362 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2ee8141690 con 0x7f2ed4075520 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ecf7fe640 10 client.20963.objecter ms_dispatch 0x7f2ee8000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ecf7fe640 10 client.20963.objecter _finish_command 2 = system:0 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ecf7fe640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 4 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ecf7fe640 15 client.20963.objecter _session_command_op_remove 0 2 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:20.831 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:20.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 10 librados: watch_flush enter 2024-09-16T04:21:20.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 10 librados: watch_flush exit 2024-09-16T04:21:20.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 10 client.20963.objecter close_session for osd.0 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6800/1243681362 conn(0x7f2ed4075520 msgr2=0x7f2ed40779c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6800/1243681362 conn(0x7f2ed4075520 0x7f2ed40779c0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 20 client.20963.objecter put_session s=0x7f2ed4074e70 osd=0 1 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 20 client.20963.objecter shutdown clearing up homeless session... 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 10 client.20963.objecter successfully canceled tick 2024-09-16T04:21:20.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6804/198622594 conn(0x7f2ed406dc00 msgr2=0x7f2ed40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 --2- 172.21.15.70:0/1388460464 >> v2:172.21.15.70:6804/198622594 conn(0x7f2ed406dc00 0x7f2ed40700c0 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009560 tx=0x7f2ed8009190 comp rx=0 tx=0).stop 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 10 monclient: shutdown 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8146e50 msgr2=0x7f2ee8143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 --2- 172.21.15.70:0/1388460464 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f2ee8146e50 0x7f2ee8143510 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80af860 tx=0x7f2edc0046c0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2ee77fe640 1 -- 172.21.15.70:0/1388460464 reap_dead start 2024-09-16T04:21:20.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 shutdown_connections 2024-09-16T04:21:20.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 >> 172.21.15.70:0/1388460464 conn(0x7f2ee80685b0 msgr2=0x7f2ee80b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:20.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 shutdown_connections 2024-09-16T04:21:20.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 -- 172.21.15.70:0/1388460464 wait complete. 2024-09-16T04:21:20.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.796+0000 7f2eed49b640 1 librados: shutdown 2024-09-16T04:21:20.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:21:20.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.938+0000 7f3d4adc4640 10 monclient: get_monmap_and_config 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.938+0000 7f3d4adc4640 10 monclient: build_initial_monmap 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.938+0000 7f3d4adc4640 10 monclient: monmap: 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:20.938786+0000 2024-09-16T04:21:20.940 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:20.938786+0000 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:20.941 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 Processor -- start 2024-09-16T04:21:20.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 -- start start 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient: init 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d440b5650 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d440b3120 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 0x7f3d4414e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d44146c10 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d440aae40 con 0x7f3d440b3120 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d44140b20 con 0x7f3d440b5650 2024-09-16T04:21:20.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3d440b5a30 con 0x7f3d44146c10 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d4adc4640 10 monclient(hunting): authenticate will time out at 2639.600703s 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d43fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3d440b3120 0x7f3d44146500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42452/0 (socket says 172.21.15.70:42452) 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 0x7f3d4414e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d43fff640 1 -- 172.21.15.70:0/3638818163 learned_addr learned my addr 172.21.15.70:0/3638818163 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): get_auth_request con 0x7f3d44146c10 auth_method 0 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.939+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 10 monclient(hunting): handle_auth_done global_id 20966 payload 274 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 1 -- 172.21.15.70:0/3638818163 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918962343 0 0) 0x7f3d440aae40 con 0x7f3d440b3120 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 1 -- 172.21.15.70:0/3638818163 >> v1:172.21.15.163:6789/0 conn(0x7f3d440b5650 legacy=0x7f3d44140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 1 -- 172.21.15.70:0/3638818163 >> v1:172.21.15.70:6789/0 conn(0x7f3d440b3120 legacy=0x7f3d44146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 10 monclient: _finish_hunting 0 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 1 monclient: found mon.noname-b 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 1 -- 172.21.15.70:0/3638818163 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d440b5cd0 con 0x7f3d44146c10 2024-09-16T04:21:20.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d49361640 1 --2- 172.21.15.70:0/3638818163 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 0x7f3d4414e8a0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009dc0 tx=0x7f3d3c02a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b7f2f6b0809e0f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 1 -- 172.21.15.70:0/3638818163 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c02ae60 con 0x7f3d44146c10 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: dump: 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.956 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.957 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 1 -- 172.21.15.70:0/3638818163 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3c007740 con 0x7f3d44146c10 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d4adc4640 5 monclient: authenticate success, global_id 20966 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d4adc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d4adc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.940+0000 7f3d437fe640 1 -- 172.21.15.70:0/3638818163 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c007b10 con 0x7f3d44146c10 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 4 set_mon_vals no callback set 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:20.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d437fe640 10 monclient: dump: 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.959 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 10 monclient: shutdown 2024-09-16T04:21:20.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 -- 172.21.15.70:0/3638818163 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 msgr2=0x7f3d4414e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/3638818163 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 0x7f3d4414e8a0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009dc0 tx=0x7f3d3c02a640 comp rx=0 tx=0).stop 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 -- 172.21.15.70:0/3638818163 shutdown_connections 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/3638818163 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d44146c10 0x7f3d4414e8a0 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 -- 172.21.15.70:0/3638818163 >> 172.21.15.70:0/3638818163 conn(0x7f3d440679c0 msgr2=0x7f3d44067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 -- 172.21.15.70:0/3638818163 shutdown_connections 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.941+0000 7f3d4adc4640 1 -- 172.21.15.70:0/3638818163 wait complete. 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 10 monclient: build_initial_monmap 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 10 monclient: monmap: 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:20.961 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:20.962 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 librados: starting msgr at 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 librados: starting objecter 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 Processor -- start 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 -- start start 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 librados: setting wanted keys 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 1 librados: calling monclient init 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.942+0000 7f3d4adc4640 10 monclient: init 2024-09-16T04:21:20.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d440b3120 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d440b5650 0x7f3d44132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d440b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 0x7f3d44133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d44146c10 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d440aab70 con 0x7f3d44146c10 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d440aa9f0 con 0x7f3d440b5650 2024-09-16T04:21:20.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f3d4414ede0 con 0x7f3d440b3120 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d4adc4640 10 monclient(hunting): authenticate will time out at 2639.604330s 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54750/0 (socket says 172.21.15.70:54750) 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 -- 172.21.15.70:0/4094608610 learned_addr learned my addr 172.21.15.70:0/4094608610 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): get_auth_request con 0x7f3d440b3120 auth_method 0 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 0x7f3d44133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): get_auth_request con 0x7f3d44146c10 auth_method 0 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d440b5650 0x7f3d44132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): get_auth_request con 0x7f3d440b5650 auth_method 0 2024-09-16T04:21:20.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient(hunting): handle_auth_done global_id 20969 payload 824 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 -- 172.21.15.70:0/4094608610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d440b5650 msgr2=0x7f3d44132ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d440b5650 0x7f3d44132ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 -- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 msgr2=0x7f3d44133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 0x7f3d44133430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:20.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient: _finish_hunting 0 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 monclient: found mon.noname-c 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 -- 172.21.15.70:0/4094608610 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d441430a0 con 0x7f3d440b3120 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d49361640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 0x7f3d44133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d48b60640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c480 tx=0x7f3d3800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d417fa640 10 client.?.objecter ms_handle_connect 0x7f3d440b3120 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.943+0000 7f3d417fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d380052d0 con 0x7f3d440b3120 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: dump: 2024-09-16T04:21:20.969 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: _finish_auth 0 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: _check_auth_tickets 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38005470 con 0x7f3d440b3120 2024-09-16T04:21:20.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 5 monclient: authenticate success, global_id 20969 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 10 monclient: _renew_subs 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d44143380 con 0x7f3d440b3120 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38012870 con 0x7f3d440b3120 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d417fa640 10 monclient: dump: 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:20.971 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 10 client.20969.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d42ffd640 4 set_mon_vals no callback set 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 10 monclient: _renew_subs 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d44143b60 con 0x7f3d440b3120 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.944+0000 7f3d4adc4640 1 librados: init done 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.945+0000 7f3d4adc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:20.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.945+0000 7f3d4adc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.945+0000 7f3d4adc4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.945+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d440b9c70 con 0x7f3d440b3120 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.945+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f3d38012a10 con 0x7f3d440b3120 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 1 --2- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 0x7f3d280700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d43fff640 1 --2- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 0x7f3d280700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d43fff640 10 monclient: get_auth_request con 0x7f3d2806dc00 auth_method 0 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f3d3808acd0 con 0x7f3d440b3120 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 10 client.20969.objecter ms_dispatch 0x7f3d44000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 3 client.20969.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 3 client.20969.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 20 client.20969.objecter dump_active .. 0 homeless 2024-09-16T04:21:20.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d417fa640 10 client.20969.objecter ms_handle_connect 0x7f3d2806dc00 2024-09-16T04:21:20.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.946+0000 7f3d43fff640 1 --2- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 0x7f3d280700c0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f3d30007f70 tx=0x7f3d30007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:20.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.950+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3d38059040 con 0x7f3d440b3120 2024-09-16T04:21:20.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.950+0000 7f3d417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:20.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:20.950+0000 7f3d417fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:21.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.116+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f3d44146500 con 0x7f3d2806dc00 2024-09-16T04:21:21.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.116+0000 7f3d417fa640 1 -- 172.21.15.70:0/4094608610 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3d44146500 con 0x7f3d2806dc00 2024-09-16T04:21:21.118 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:21:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 10 librados: watch_flush enter 2024-09-16T04:21:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 10 librados: watch_flush exit 2024-09-16T04:21:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 20 client.20969.objecter shutdown clearing up homeless session... 2024-09-16T04:21:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 10 client.20969.objecter successfully canceled tick 2024-09-16T04:21:21.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 msgr2=0x7f3d280700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 0x7f3d280700c0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f3d30007f70 tx=0x7f3d30007f00 comp rx=0 tx=0).stop 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 10 monclient: shutdown 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 msgr2=0x7f3d441420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800c480 tx=0x7f3d3800c940 comp rx=0 tx=0).stop 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 shutdown_connections 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> v2:172.21.15.70:6804/198622594 conn(0x7f3d2806dc00 0x7f3d280700c0 unknown :-1 s=CLOSED pgs=4029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d44146c10 0x7f3d44133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d440b5650 0x7f3d44132ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 --2- 172.21.15.70:0/4094608610 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f3d440b3120 0x7f3d441420b0 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 >> 172.21.15.70:0/4094608610 conn(0x7f3d440679c0 msgr2=0x7f3d440b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 shutdown_connections 2024-09-16T04:21:21.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 -- 172.21.15.70:0/4094608610 wait complete. 2024-09-16T04:21:21.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.118+0000 7f3d4adc4640 1 librados: shutdown 2024-09-16T04:21:21.130 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:21:21.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.229+0000 7efe66253640 10 monclient: get_monmap_and_config 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.229+0000 7efe66253640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.229+0000 7efe66253640 10 monclient: monmap: 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:21.230161+0000 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:21.230161+0000 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:21.231 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:21.232 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:21.232 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.230+0000 7efe66253640 1 Processor -- start 2024-09-16T04:21:21.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.230+0000 7efe66253640 1 -- start start 2024-09-16T04:21:21.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.230+0000 7efe66253640 10 monclient: init 2024-09-16T04:21:21.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efe6006c1e0 0x7efe6006c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): picked mon.noname-a con 0x7efe6006c1e0 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7efe60140200 0x7efe60140650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): picked mon.noname-c con 0x7efe60140200 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe60143160 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe600aef50 con 0x7efe60143160 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efe600aed00 con 0x7efe6006c1e0 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7efe600b0df0 con 0x7efe60140200 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe66253640 10 monclient(hunting): authenticate will time out at 2639.892137s 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efe6006c1e0 0x7efe6006c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efe6006c1e0 0x7efe6006c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33326/0 (socket says 172.21.15.70:33326) 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 -- 172.21.15.70:0/3741809443 learned_addr learned my addr 172.21.15.70:0/3741809443 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): get_auth_request con 0x7efe6006c1e0 auth_method 0 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5effd640 1 --2- 172.21.15.70:0/3741809443 >> v2:172.21.15.163:3300/0 conn(0x7efe60140200 0x7efe60140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5effd640 10 monclient(hunting): get_auth_request con 0x7efe60140200 auth_method 0 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_done global_id 26751 payload 274 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 -- 172.21.15.70:0/3741809443 >> v2:172.21.15.163:3300/0 conn(0x7efe60140200 msgr2=0x7efe60140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 --2- 172.21.15.70:0/3741809443 >> v2:172.21.15.163:3300/0 conn(0x7efe60140200 0x7efe60140650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 -- 172.21.15.70:0/3741809443 >> v1:172.21.15.163:6789/0 conn(0x7efe60143160 legacy=0x7efe60143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.231+0000 7efe5f7fe640 1 -- 172.21.15.70:0/3741809443 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe60140b90 con 0x7efe6006c1e0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5f7fe640 1 --2- 172.21.15.70:0/3741809443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6006c1e0 0x7efe6006c5c0 secure :-1 s=READY pgs=5738 cs=0 l=1 rev1=1 crypto rx=0x7efe6006f330 tx=0x7efe5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0582e129924fb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 1 -- 172.21.15.70:0/3741809443 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50037070 con 0x7efe6006c1e0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:21.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: dump: 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 1 -- 172.21.15.70:0/3741809443 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe50032440 con 0x7efe6006c1e0 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 5 monclient: authenticate success, global_id 26751 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 1 -- 172.21.15.70:0/3741809443 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50032810 con 0x7efe6006c1e0 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 4 set_mon_vals no callback set 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe5e7fc640 10 monclient: dump: 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.245 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 10 monclient: shutdown 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 1 -- 172.21.15.70:0/3741809443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6006c1e0 msgr2=0x7efe6006c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.232+0000 7efe66253640 1 --2- 172.21.15.70:0/3741809443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6006c1e0 0x7efe6006c5c0 secure :-1 s=READY pgs=5738 cs=0 l=1 rev1=1 crypto rx=0x7efe6006f330 tx=0x7efe5002a1e0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 -- 172.21.15.70:0/3741809443 shutdown_connections 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 --2- 172.21.15.70:0/3741809443 >> v2:172.21.15.163:3300/0 conn(0x7efe60140200 0x7efe60140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 --2- 172.21.15.70:0/3741809443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6006c1e0 0x7efe6006c5c0 unknown :-1 s=CLOSED pgs=5738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 -- 172.21.15.70:0/3741809443 >> 172.21.15.70:0/3741809443 conn(0x7efe600b6de0 msgr2=0x7efe600b71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 -- 172.21.15.70:0/3741809443 shutdown_connections 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 -- 172.21.15.70:0/3741809443 wait complete. 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 10 monclient: monmap: 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 librados: starting msgr at 2024-09-16T04:21:21.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 librados: starting objecter 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.233+0000 7efe66253640 1 Processor -- start 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 -- start start 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 librados: setting wanted keys 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 librados: calling monclient init 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient: init 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe6006c1e0 0x7efe601db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): picked mon.noname-b con 0x7efe6006c1e0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 0x7efe60145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): picked mon.noname-a con 0x7efe60140200 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efe60143160 0x7efe60145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): picked mon.noname-c con 0x7efe60143160 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe60143e00 con 0x7efe60140200 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efe60143c80 con 0x7efe6006c1e0 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7efe6014d800 con 0x7efe60143160 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe66253640 10 monclient(hunting): authenticate will time out at 2639.895682s 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 0x7efe60145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efe60143160 0x7efe60145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 0x7efe60145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33342/0 (socket says 172.21.15.70:33342) 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 1 -- 172.21.15.70:0/3609378123 learned_addr learned my addr 172.21.15.70:0/3609378123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): get_auth_request con 0x7efe60140200 auth_method 0 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): get_auth_request con 0x7efe60143160 auth_method 0 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5f7fe640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe6006c1e0 0x7efe601db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5f7fe640 10 monclient(hunting): get_auth_request con 0x7efe6006c1e0 auth_method 0 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.234+0000 7efe5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 10 monclient(hunting): handle_auth_done global_id 26754 payload 824 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 -- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efe60143160 msgr2=0x7efe60145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efe60143160 0x7efe60145e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 -- 172.21.15.70:0/3609378123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe6006c1e0 msgr2=0x7efe601db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe6006c1e0 0x7efe601db9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 monclient: found mon.noname-a 2024-09-16T04:21:21.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 -- 172.21.15.70:0/3609378123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe60147650 con 0x7efe60140200 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5ffff640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7efe60143160 0x7efe60145e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5effd640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 0x7efe60145640 secure :-1 s=READY pgs=5739 cs=0 l=1 rev1=1 crypto rx=0x7efe5400d800 tx=0x7efe5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 client.?.objecter ms_handle_connect 0x7efe60140200 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400b6c0 con 0x7efe60140200 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: dump: 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.252 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe54019070 con 0x7efe60140200 2024-09-16T04:21:21.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 5 monclient: authenticate success, global_id 26754 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 10 monclient: _renew_subs 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe60073660 con 0x7efe60140200 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400ba80 con 0x7efe60140200 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe47fff640 10 monclient: dump: 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.254 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 10 client.26754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 10 monclient: _renew_subs 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe5dffb640 4 set_mon_vals no callback set 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe60146cf0 con 0x7efe60140200 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.235+0000 7efe66253640 1 librados: init done 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 10 client.26754.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 15 client.26754.objecter _session_command_op_assign -1 1 2024-09-16T04:21:21.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 20 client.26754.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 10 client.26754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efe6006c5c0 con 0x7efe60140200 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.236+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7efe5400bc20 con 0x7efe60140200 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6804/198622594 conn(0x7efe4c06dc30 0x7efe4c0700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5f7fe640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6804/198622594 conn(0x7efe4c06dc30 0x7efe4c0700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7efe5408b4e0 con 0x7efe60140200 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 client.26754.objecter ms_dispatch 0x7efe60000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 3 client.26754.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:21.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 3 client.26754.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5f7fe640 10 monclient: get_auth_request con 0x7efe4c06dc30 auth_method 0 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 client.26754.objecter checking command 1 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6813/1499120614 conn(0x7efe4c075550 0x7efe4c0779f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter _get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 15 client.26754.objecter _session_command_op_remove -1 1 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter _get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 15 client.26754.objecter _session_command_op_assign 1 1 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 4 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 client.26754.objecter _send_command 1 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 --> v2:172.21.15.70:6813/1499120614 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe4c078080 con 0x7efe4c075550 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 20 client.26754.objecter dump_active .. 0 homeless 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7efe54092050 con 0x7efe60140200 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5ffff640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6813/1499120614 conn(0x7efe4c075550 0x7efe4c0779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5ffff640 10 monclient: get_auth_request con 0x7efe4c075550 auth_method 0 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 client.26754.objecter ms_handle_connect 0x7efe4c06dc30 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5f7fe640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6804/198622594 conn(0x7efe4c06dc30 0x7efe4c0700f0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7efe50034000 tx=0x7efe5002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe47fff640 10 client.26754.objecter ms_handle_connect 0x7efe4c075550 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.237+0000 7efe5ffff640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6813/1499120614 conn(0x7efe4c075550 0x7efe4c0779f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.238+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== osd.1 v2:172.21.15.70:6813/1499120614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe4c078080 con 0x7efe4c075550 2024-09-16T04:21:21.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.238+0000 7efe47fff640 10 client.26754.objecter ms_dispatch 0x7efe60000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:21.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.238+0000 7efe47fff640 10 client.26754.objecter _finish_command 1 = system:0 2024-09-16T04:21:21.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.238+0000 7efe47fff640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 4 2024-09-16T04:21:21.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.238+0000 7efe47fff640 15 client.26754.objecter _session_command_op_remove 1 1 2024-09-16T04:21:21.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 10 client.26754.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 15 client.26754.objecter _session_command_op_assign -1 2 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 20 client.26754.objecter _get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 20 client.26754.objecter _get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 15 client.26754.objecter _session_command_op_remove -1 2 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 20 client.26754.objecter get_session s=0x7efe4c074ea0 osd=1 3 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 15 client.26754.objecter _session_command_op_assign 1 2 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 4 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 10 client.26754.objecter _send_command 2 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.250+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 --> v2:172.21.15.70:6813/1499120614 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efe6014d5e0 con 0x7efe4c075550 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe47fff640 1 -- 172.21.15.70:0/3609378123 <== osd.1 v2:172.21.15.70:6813/1499120614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efe6014d5e0 con 0x7efe4c075550 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe47fff640 10 client.26754.objecter ms_dispatch 0x7efe60000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:21.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe47fff640 10 client.26754.objecter _finish_command 2 = system:0 2024-09-16T04:21:21.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe47fff640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 4 2024-09-16T04:21:21.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe47fff640 15 client.26754.objecter _session_command_op_remove 1 2 2024-09-16T04:21:21.261 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:21.261 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:21.261 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 10 librados: watch_flush enter 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 10 librados: watch_flush exit 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 10 client.26754.objecter close_session for osd.1 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6813/1499120614 conn(0x7efe4c075550 msgr2=0x7efe4c0779f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6813/1499120614 conn(0x7efe4c075550 0x7efe4c0779f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 20 client.26754.objecter put_session s=0x7efe4c074ea0 osd=1 1 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 20 client.26754.objecter shutdown clearing up homeless session... 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 10 client.26754.objecter successfully canceled tick 2024-09-16T04:21:21.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6804/198622594 conn(0x7efe4c06dc30 msgr2=0x7efe4c0700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 --2- 172.21.15.70:0/3609378123 >> v2:172.21.15.70:6804/198622594 conn(0x7efe4c06dc30 0x7efe4c0700f0 secure :-1 s=READY pgs=4030 cs=0 l=1 rev1=1 crypto rx=0x7efe50034000 tx=0x7efe5002e000 comp rx=0 tx=0).stop 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 10 monclient: shutdown 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 msgr2=0x7efe60145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 --2- 172.21.15.70:0/3609378123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe60140200 0x7efe60145640 secure :-1 s=READY pgs=5739 cs=0 l=1 rev1=1 crypto rx=0x7efe5400d800 tx=0x7efe5400dcc0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe5ffff640 1 -- 172.21.15.70:0/3609378123 reap_dead start 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 shutdown_connections 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 >> 172.21.15.70:0/3609378123 conn(0x7efe600b6de0 msgr2=0x7efe6013ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 shutdown_connections 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 -- 172.21.15.70:0/3609378123 wait complete. 2024-09-16T04:21:21.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.251+0000 7efe66253640 1 librados: shutdown 2024-09-16T04:21:21.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:21:21.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.361+0000 7eff828a4640 10 monclient: get_monmap_and_config 2024-09-16T04:21:21.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.361+0000 7eff828a4640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.361+0000 7eff828a4640 10 monclient: monmap: 2024-09-16T04:21:21.362 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.362 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:21.361805+0000 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:21.361805+0000 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:21.363 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 Processor -- start 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 -- start start 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient: init 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-b con 0x7eff7c0b5650 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff7c140410 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7eff7c140830 0x7eff7c146c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-c con 0x7eff7c140830 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7c0aae40 con 0x7eff7c140410 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7eff7c0b5a30 con 0x7eff7c140830 2024-09-16T04:21:21.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c147150 con 0x7eff7c0b5650 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff828a4640 10 monclient(hunting): authenticate will time out at 2640.023785s 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54780/0 (socket says 172.21.15.70:54780) 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 1 -- 172.21.15.70:0/2698775325 learned_addr learned my addr 172.21.15.70:0/2698775325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 10 monclient(hunting): get_auth_request con 0x7eff7c0b5650 auth_method 0 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.362+0000 7eff7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient(hunting): handle_auth_done global_id 20975 payload 274 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff80e41640 1 --2- 172.21.15.70:0/2698775325 >> v2:172.21.15.163:3300/0 conn(0x7eff7c140830 0x7eff7c146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 -- 172.21.15.70:0/2698775325 >> v2:172.21.15.163:3300/0 conn(0x7eff7c140830 msgr2=0x7eff7c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 --2- 172.21.15.70:0/2698775325 >> v2:172.21.15.163:3300/0 conn(0x7eff7c140830 0x7eff7c146c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 -- 172.21.15.70:0/2698775325 >> v1:172.21.15.163:6789/0 conn(0x7eff7c140410 legacy=0x7eff7c146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 monclient: found mon.noname-b 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 -- 172.21.15.70:0/2698775325 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c0b5cd0 con 0x7eff7c0b5650 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7bfff640 1 --2- 172.21.15.70:0/2698775325 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009870 tx=0x7eff6c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff1004fc6d327e41 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7affd640 1 -- 172.21.15.70:0/2698775325 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c037070 con 0x7eff7c0b5650 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.363+0000 7eff7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: dump: 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 5 monclient: authenticate success, global_id 20975 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 1 -- 172.21.15.70:0/2698775325 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c032440 con 0x7eff7c0b5650 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 4 set_mon_vals no callback set 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 1 -- 172.21.15.70:0/2698775325 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c032810 con 0x7eff7c0b5650 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff7affd640 10 monclient: dump: 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.371 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 10 monclient: shutdown 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 -- 172.21.15.70:0/2698775325 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 msgr2=0x7eff7c0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 --2- 172.21.15.70:0/2698775325 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009870 tx=0x7eff6c02a180 comp rx=0 tx=0).stop 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 -- 172.21.15.70:0/2698775325 shutdown_connections 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 --2- 172.21.15.70:0/2698775325 >> v2:172.21.15.163:3300/0 conn(0x7eff7c140830 0x7eff7c146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 --2- 172.21.15.70:0/2698775325 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c0b5650 0x7eff7c0b3120 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 -- 172.21.15.70:0/2698775325 >> 172.21.15.70:0/2698775325 conn(0x7eff7c0679c0 msgr2=0x7eff7c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 -- 172.21.15.70:0/2698775325 shutdown_connections 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.364+0000 7eff828a4640 1 -- 172.21.15.70:0/2698775325 wait complete. 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 10 monclient: monmap: 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:21.373 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 librados: starting msgr at 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 librados: starting objecter 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 Processor -- start 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 -- start start 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 librados: setting wanted keys 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 1 librados: calling monclient init 2024-09-16T04:21:21.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.365+0000 7eff828a4640 10 monclient: init 2024-09-16T04:21:21.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:21:21.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c0b5650 0x7eff7c14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-b con 0x7eff7c0b5650 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-c con 0x7eff7c140410 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 0x7eff7c147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): picked mon.noname-a con 0x7eff7c140830 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c14da10 con 0x7eff7c140830 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c14d890 con 0x7eff7c0b5650 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c14db90 con 0x7eff7c140410 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff828a4640 10 monclient(hunting): authenticate will time out at 2640.027359s 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 0x7eff7c147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54794/0 (socket says 172.21.15.70:54794) 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 -- 172.21.15.70:0/3531645695 learned_addr learned my addr 172.21.15.70:0/3531645695 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): get_auth_request con 0x7eff7c140410 auth_method 0 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): get_auth_request con 0x7eff7c140830 auth_method 0 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c0b5650 0x7eff7c14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): get_auth_request con 0x7eff7c0b5650 auth_method 0 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient(hunting): handle_auth_done global_id 20978 payload 824 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 -- 172.21.15.70:0/3531645695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c0b5650 msgr2=0x7eff7c14a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c0b5650 0x7eff7c14a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 -- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 msgr2=0x7eff7c147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 0x7eff7c147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 -- 172.21.15.70:0/3531645695 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c14c200 con 0x7eff7c140410 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff80e41640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 0x7eff7c147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff7b7fe640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c430 tx=0x7eff7000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff78ff9640 10 client.?.objecter ms_handle_connect 0x7eff7c140410 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.366+0000 7eff78ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70017070 con 0x7eff7c140410 2024-09-16T04:21:21.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: dump: 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff700035f0 con 0x7eff7c140410 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 5 monclient: authenticate success, global_id 20978 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 10 monclient: _renew_subs 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff700039e0 con 0x7eff7c140410 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c14c4e0 con 0x7eff7c140410 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff78ff9640 10 monclient: dump: 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.384 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff7a7fc640 4 set_mon_vals no callback set 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 10 client.20978.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 10 monclient: _renew_subs 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c14abf0 con 0x7eff7c140410 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.367+0000 7eff828a4640 1 librados: init done 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff828a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff828a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff828a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7c0b3120 con 0x7eff7c140410 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7eff7001a070 con 0x7eff7c140410 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.368+0000 7eff78ff9640 1 --2- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 0x7eff680700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7eff7008a550 con 0x7eff7c140410 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 10 client.20978.objecter ms_dispatch 0x7eff7c000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 3 client.20978.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 3 client.20978.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff7bfff640 1 --2- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 0x7eff680700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff7bfff640 10 monclient: get_auth_request con 0x7eff6806dc00 auth_method 0 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 20 client.20978.objecter dump_active .. 0 homeless 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff78ff9640 10 client.20978.objecter ms_handle_connect 0x7eff6806dc00 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.369+0000 7eff7bfff640 1 --2- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 0x7eff680700c0 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7eff6c034000 tx=0x7eff6c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.373+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff700588c0 con 0x7eff7c140410 2024-09-16T04:21:21.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.373+0000 7eff78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.373+0000 7eff78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:21.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.541+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7eff7c145d90 con 0x7eff6806dc00 2024-09-16T04:21:21.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.542+0000 7eff78ff9640 1 -- 172.21.15.70:0/3531645695 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7eff7c145d90 con 0x7eff6806dc00 2024-09-16T04:21:21.543 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 10 librados: watch_flush enter 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 10 librados: watch_flush exit 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 20 client.20978.objecter shutdown clearing up homeless session... 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 10 client.20978.objecter successfully canceled tick 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 msgr2=0x7eff680700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 0x7eff680700c0 secure :-1 s=READY pgs=4031 cs=0 l=1 rev1=1 crypto rx=0x7eff6c034000 tx=0x7eff6c02e000 comp rx=0 tx=0).stop 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 10 monclient: shutdown 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 msgr2=0x7eff7c147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c430 tx=0x7eff7000c8f0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 shutdown_connections 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> v2:172.21.15.70:6804/198622594 conn(0x7eff6806dc00 0x7eff680700c0 unknown :-1 s=CLOSED pgs=4031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff7c140830 0x7eff7c147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7eff7c140410 0x7eff7c147aa0 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 --2- 172.21.15.70:0/3531645695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff7c0b5650 0x7eff7c14a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 >> 172.21.15.70:0/3531645695 conn(0x7eff7c0679c0 msgr2=0x7eff7c0b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 shutdown_connections 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 -- 172.21.15.70:0/3531645695 wait complete. 2024-09-16T04:21:21.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.544+0000 7eff828a4640 1 librados: shutdown 2024-09-16T04:21:21.556 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:21:21.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.655+0000 7f761e1a6640 10 monclient: get_monmap_and_config 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.655+0000 7f761e1a6640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.655+0000 7f761e1a6640 10 monclient: monmap: 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:21.656293+0000 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:21.656293+0000 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.657 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:21.658 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:21.658 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.656+0000 7f761e1a6640 1 Processor -- start 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.656+0000 7f761e1a6640 1 -- start start 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.656+0000 7f761e1a6640 10 monclient: init 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 0x7f761813ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-c con 0x7f76180b5e80 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7618140490 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 0x7f761814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-b con 0x7f7618146bb0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76180abcf0 con 0x7f7618140490 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f76180b6260 con 0x7f76180b5e80 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f76181408c0 con 0x7f7618146bb0 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f761e1a6640 10 monclient(hunting): authenticate will time out at 2640.318304s 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7616ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7618140490 0x7f76181464a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42464/0 (socket says 172.21.15.70:42464) 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 --2- 172.21.15.70:0/77723162 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 0x7f761814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7616ffd640 1 -- 172.21.15.70:0/77723162 learned_addr learned my addr 172.21.15.70:0/77723162 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): get_auth_request con 0x7f7618146bb0 auth_method 0 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient(hunting): handle_auth_done global_id 20981 payload 274 2024-09-16T04:21:21.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f76177fe640 1 --2- 172.21.15.70:0/77723162 >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 0x7f761813ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 -- 172.21.15.70:0/77723162 >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 msgr2=0x7f761813ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 --2- 172.21.15.70:0/77723162 >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 0x7f761813ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 -- 172.21.15.70:0/77723162 >> v1:172.21.15.70:6789/0 conn(0x7f7618140490 legacy=0x7f76181464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 monclient: found mon.noname-b 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f76167fc640 1 -- 172.21.15.70:0/77723162 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2560979885 0 0) 0x7f76180abcf0 con 0x7f7618140490 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f7617fff640 1 -- 172.21.15.70:0/77723162 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76180abf60 con 0x7f7618146bb0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f76177fe640 10 monclient: get_auth_request con 0x7f76180b5e80 auth_method 0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f76177fe640 1 --2- 172.21.15.70:0/77723162 >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 0x7f761813ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.657+0000 7f76167fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f7617fff640 1 --2- 172.21.15.70:0/77723162 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 0x7f761814d7a0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f7610009d90 tx=0x7f761002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b8bd511048ed69 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 1 -- 172.21.15.70:0/77723162 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7610031600 con 0x7f7618146bb0 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: dump: 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 1 -- 172.21.15.70:0/77723162 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76100317a0 con 0x7f7618146bb0 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f761e1a6640 5 monclient: authenticate success, global_id 20981 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f761e1a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f761e1a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 1 -- 172.21.15.70:0/77723162 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7610031b70 con 0x7f7618146bb0 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f761e1a6640 4 set_mon_vals no callback set 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.658+0000 7f76167fc640 10 monclient: dump: 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.671 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 10 monclient: shutdown 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 -- 172.21.15.70:0/77723162 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 msgr2=0x7f761814d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 --2- 172.21.15.70:0/77723162 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 0x7f761814d7a0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f7610009d90 tx=0x7f761002a5d0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 -- 172.21.15.70:0/77723162 shutdown_connections 2024-09-16T04:21:21.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 --2- 172.21.15.70:0/77723162 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f7618146bb0 0x7f761814d7a0 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 --2- 172.21.15.70:0/77723162 >> v2:172.21.15.163:3300/0 conn(0x7f76180b5e80 0x7f761813ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 -- 172.21.15.70:0/77723162 >> 172.21.15.70:0/77723162 conn(0x7f761806cbb0 msgr2=0x7f761806cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 -- 172.21.15.70:0/77723162 shutdown_connections 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 -- 172.21.15.70:0/77723162 wait complete. 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 10 monclient: monmap: 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:21.673 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 librados: starting msgr at 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.659+0000 7f761e1a6640 1 librados: starting objecter 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 Processor -- start 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 -- start start 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 librados: setting wanted keys 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 librados: calling monclient init 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient: init 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 0x7f76180b9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-c con 0x7f76180b5e80 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7618140490 0x7f76181d86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-b con 0x7f7618140490 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7618146bb0 0x7f76181d8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): picked mon.noname-a con 0x7f7618146bb0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76181471a0 con 0x7f7618146bb0 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7618147020 con 0x7f7618140490 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f76180aac60 con 0x7f76180b5e80 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f761e1a6640 10 monclient(hunting): authenticate will time out at 2640.321782s 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 0x7f76180b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 0x7f76180b9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54810/0 (socket says 172.21.15.70:54810) 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 1 -- 172.21.15.70:0/2654599059 learned_addr learned my addr 172.21.15.70:0/2654599059 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): get_auth_request con 0x7f76180b5e80 auth_method 0 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7618146bb0 0x7f76181d8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): get_auth_request con 0x7f7618146bb0 auth_method 0 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f76177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.660+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7616ffd640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7618140490 0x7f76181d86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7616ffd640 10 monclient(hunting): get_auth_request con 0x7f7618140490 auth_method 0 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7616ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7616ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 10 monclient(hunting): handle_auth_done global_id 20984 payload 824 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 -- 172.21.15.70:0/2654599059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7618140490 msgr2=0x7f76181d86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7618140490 0x7f76181d86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 -- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7618146bb0 msgr2=0x7f76181d8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7618146bb0 0x7f76181d8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 monclient: found mon.noname-c 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 -- 172.21.15.70:0/2654599059 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76180ad760 con 0x7f76180b5e80 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f76177fe640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 0x7f76180b9470 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f760800c7f0 tx=0x7f760800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 client.?.objecter ms_handle_connect 0x7f76180b5e80 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7608015070 con 0x7f76180b5e80 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: dump: 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.678 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7608007a30 con 0x7f76180b5e80 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 5 monclient: authenticate success, global_id 20984 2024-09-16T04:21:21.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 10 monclient: _renew_subs 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76180b25e0 con 0x7f76180b5e80 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7608007e00 con 0x7f76180b5e80 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f75fffff640 10 monclient: dump: 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.680 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 10 client.20984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 10 monclient: _renew_subs 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76180ac740 con 0x7f76180b5e80 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f761e1a6640 1 librados: init done 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.661+0000 7f7615ffb640 4 set_mon_vals no callback set 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 10 client.20984.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 15 client.20984.objecter _session_command_op_assign -1 1 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 20 client.20984.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 10 client.20984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:21.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f761813ff50 con 0x7f76180b5e80 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.662+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f7608018070 con 0x7f76180b5e80 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6804/198622594 conn(0x7f760406dc00 0x7f76040700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7616ffd640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6804/198622594 conn(0x7f760406dc00 0x7f76040700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f760808b130 con 0x7f76180b5e80 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 client.20984.objecter ms_dispatch 0x7f7618000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 3 client.20984.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 3 client.20984.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7616ffd640 10 monclient: get_auth_request con 0x7f760406dc00 auth_method 0 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 client.20984.objecter checking command 1 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6809/2584301924 conn(0x7f7604075580 0x7f7604077a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter _get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 15 client.20984.objecter _session_command_op_remove -1 1 2024-09-16T04:21:21.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter _get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 15 client.20984.objecter _session_command_op_assign 2 1 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 4 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 client.20984.objecter _send_command 1 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 --> v2:172.21.15.70:6809/2584301924 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f760801ce50 con 0x7f7604075580 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 20 client.20984.objecter dump_active .. 0 homeless 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7f7608092050 con 0x7f76180b5e80 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7617fff640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6809/2584301924 conn(0x7f7604075580 0x7f7604077a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7617fff640 10 monclient: get_auth_request con 0x7f7604075580 auth_method 0 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 client.20984.objecter ms_handle_connect 0x7f760406dc00 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7616ffd640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6804/198622594 conn(0x7f760406dc00 0x7f76040700c0 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7f76181dcc70 tx=0x7f760c007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f75fffff640 10 client.20984.objecter ms_handle_connect 0x7f7604075580 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.663+0000 7f7617fff640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6809/2584301924 conn(0x7f7604075580 0x7f7604077a20 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.664+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== osd.2 v2:172.21.15.70:6809/2584301924 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f760801ce50 con 0x7f7604075580 2024-09-16T04:21:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.664+0000 7f75fffff640 10 client.20984.objecter ms_dispatch 0x7f7618000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.664+0000 7f75fffff640 10 client.20984.objecter _finish_command 1 = system:0 2024-09-16T04:21:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.664+0000 7f75fffff640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 4 2024-09-16T04:21:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.664+0000 7f75fffff640 15 client.20984.objecter _session_command_op_remove 2 1 2024-09-16T04:21:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 10 client.20984.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 15 client.20984.objecter _session_command_op_assign -1 2 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 20 client.20984.objecter _get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 20 client.20984.objecter _get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 15 client.20984.objecter _session_command_op_remove -1 2 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 20 client.20984.objecter get_session s=0x7f7604074ed0 osd=2 3 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 15 client.20984.objecter _session_command_op_assign 2 2 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 4 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 10 client.20984.objecter _send_command 2 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.676+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 --> v2:172.21.15.70:6809/2584301924 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f761806d300 con 0x7f7604075580 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f75fffff640 1 -- 172.21.15.70:0/2654599059 <== osd.2 v2:172.21.15.70:6809/2584301924 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f761806d300 con 0x7f7604075580 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f75fffff640 10 client.20984.objecter ms_dispatch 0x7f7618000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f75fffff640 10 client.20984.objecter _finish_command 2 = system:0 2024-09-16T04:21:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f75fffff640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 4 2024-09-16T04:21:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f75fffff640 15 client.20984.objecter _session_command_op_remove 2 2 2024-09-16T04:21:21.686 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:21.686 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:21.686 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 10 librados: watch_flush enter 2024-09-16T04:21:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 10 librados: watch_flush exit 2024-09-16T04:21:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 10 client.20984.objecter close_session for osd.2 2024-09-16T04:21:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6809/2584301924 conn(0x7f7604075580 msgr2=0x7f7604077a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6809/2584301924 conn(0x7f7604075580 0x7f7604077a20 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 20 client.20984.objecter put_session s=0x7f7604074ed0 osd=2 1 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 20 client.20984.objecter shutdown clearing up homeless session... 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 10 client.20984.objecter successfully canceled tick 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6804/198622594 conn(0x7f760406dc00 msgr2=0x7f76040700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 --2- 172.21.15.70:0/2654599059 >> v2:172.21.15.70:6804/198622594 conn(0x7f760406dc00 0x7f76040700c0 secure :-1 s=READY pgs=4032 cs=0 l=1 rev1=1 crypto rx=0x7f76181dcc70 tx=0x7f760c007b50 comp rx=0 tx=0).stop 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 10 monclient: shutdown 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 msgr2=0x7f76180b9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 --2- 172.21.15.70:0/2654599059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f76180b5e80 0x7f76180b9470 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f760800c7f0 tx=0x7f760800cbf0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f7617fff640 1 -- 172.21.15.70:0/2654599059 reap_dead start 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 shutdown_connections 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 >> 172.21.15.70:0/2654599059 conn(0x7f761806cbb0 msgr2=0x7f76180b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 shutdown_connections 2024-09-16T04:21:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 -- 172.21.15.70:0/2654599059 wait complete. 2024-09-16T04:21:21.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.677+0000 7f761e1a6640 1 librados: shutdown 2024-09-16T04:21:21.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.786+0000 7f779d8a4640 10 monclient: get_monmap_and_config 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.786+0000 7f779d8a4640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.786+0000 7f779d8a4640 10 monclient: monmap: 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:21.787322+0000 2024-09-16T04:21:21.788 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:21.787322+0000 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.787+0000 7f779d8a4640 1 Processor -- start 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.787+0000 7f779d8a4640 1 -- start start 2024-09-16T04:21:21.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.787+0000 7f779d8a4640 10 monclient: init 2024-09-16T04:21:21.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f77980b5470 0x7f77980b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-a con 0x7f77980b5470 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f779813cbd0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f779813d020 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77980ac040 con 0x7f779813cbd0 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77980ac2b0 con 0x7f779813d020 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f779813d450 con 0x7f77980b5470 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f779d8a4640 10 monclient(hunting): authenticate will time out at 2640.449168s 2024-09-16T04:21:21.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f77967fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f779813cbd0 0x7f7798143040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42476/0 (socket says 172.21.15.70:42476) 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f77980b5470 0x7f77980b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f77967fc640 1 -- 172.21.15.70:0/2839119934 learned_addr learned my addr 172.21.15.70:0/2839119934 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): get_auth_request con 0x7f77980b5470 auth_method 0 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient(hunting): handle_auth_done global_id 26766 payload 274 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 1 -- 172.21.15.70:0/2839119934 >> v1:172.21.15.163:6789/0 conn(0x7f779813d020 legacy=0x7f7798143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7795ffb640 1 -- 172.21.15.70:0/2839119934 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879413401 0 0) 0x7f77980ac040 con 0x7f779813cbd0 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 1 -- 172.21.15.70:0/2839119934 >> v1:172.21.15.70:6789/0 conn(0x7f779813cbd0 legacy=0x7f7798143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 1 monclient: found mon.noname-a 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7796ffd640 1 -- 172.21.15.70:0/2839119934 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77980aaef0 con 0x7f77980b5470 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.788+0000 7f7795ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7796ffd640 1 --2- 172.21.15.70:0/2839119934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f77980b5850 secure :-1 s=READY pgs=5742 cs=0 l=1 rev1=1 crypto rx=0x7f779813d5d0 tx=0x7f779002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2013faa3ff08c1f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 1 -- 172.21.15.70:0/2839119934 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790007740 con 0x7f77980b5470 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: dump: 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.800 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 5 monclient: authenticate success, global_id 26766 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 1 -- 172.21.15.70:0/2839119934 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77900078e0 con 0x7f77980b5470 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 4 set_mon_vals no callback set 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 1 -- 172.21.15.70:0/2839119934 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790007cb0 con 0x7f77980b5470 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f7795ffb640 10 monclient: dump: 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.802 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 10 monclient: shutdown 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 1 -- 172.21.15.70:0/2839119934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 msgr2=0x7f77980b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.789+0000 7f779d8a4640 1 --2- 172.21.15.70:0/2839119934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f77980b5850 secure :-1 s=READY pgs=5742 cs=0 l=1 rev1=1 crypto rx=0x7f779813d5d0 tx=0x7f779002a6f0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 -- 172.21.15.70:0/2839119934 shutdown_connections 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 --2- 172.21.15.70:0/2839119934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f77980b5850 unknown :-1 s=CLOSED pgs=5742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 -- 172.21.15.70:0/2839119934 >> 172.21.15.70:0/2839119934 conn(0x7f77980679c0 msgr2=0x7f7798067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 -- 172.21.15.70:0/2839119934 shutdown_connections 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 -- 172.21.15.70:0/2839119934 wait complete. 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 10 monclient: build_initial_monmap 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 10 monclient: monmap: 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:21.803 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 librados: starting msgr at 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 librados: starting objecter 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.790+0000 7f779d8a4640 1 Processor -- start 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 -- start start 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 librados: setting wanted keys 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 librados: calling monclient init 2024-09-16T04:21:21.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient: init 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-a con 0x7f77980b5470 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 0x7f7798132c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-b con 0x7f779813cbd0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f779813d020 0x7f7798133170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): picked mon.noname-c con 0x7f779813d020 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77980b5f10 con 0x7f77980b5470 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77980b5d90 con 0x7f779813cbd0 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:21.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f779814e8d0 con 0x7f779813d020 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f779d8a4640 10 monclient(hunting): authenticate will time out at 2640.452702s 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33382/0 (socket says 172.21.15.70:33382) 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 1 -- 172.21.15.70:0/410543504 learned_addr learned my addr 172.21.15.70:0/410543504 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): get_auth_request con 0x7f77980b5470 auth_method 0 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f779813d020 0x7f7798133170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): get_auth_request con 0x7f779813d020 auth_method 0 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:21.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f7796ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.791+0000 7f77977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f77967fc640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 0x7f7798132c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 10 monclient(hunting): handle_auth_done global_id 26772 payload 824 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 -- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f779813d020 msgr2=0x7f7798133170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f779813d020 0x7f7798133170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 -- 172.21.15.70:0/410543504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 msgr2=0x7f7798132c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 0x7f7798132c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 monclient: found mon.noname-a 2024-09-16T04:21:21.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 -- 172.21.15.70:0/410543504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779813f510 con 0x7f77980b5470 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f77967fc640 10 monclient: get_auth_request con 0x7f779813cbd0 auth_method 0 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f77967fc640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 0x7f7798132c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f7796ffd640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 secure :-1 s=READY pgs=5743 cs=0 l=1 rev1=1 crypto rx=0x7f7790000f80 tx=0x7f7790004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 client.?.objecter ms_handle_connect 0x7f77980b5470 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790040070 con 0x7f77980b5470 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: dump: 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.808 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: _finish_auth 0 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7790031d00 con 0x7f77980b5470 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 5 monclient: authenticate success, global_id 26772 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 10 monclient: _renew_subs 2024-09-16T04:21:21.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779813f7f0 con 0x7f77980b5470 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790003bb0 con 0x7f77980b5470 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f778ffff640 10 monclient: dump: 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:21.810 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 10 client.26772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 10 monclient: _renew_subs 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f77957fa640 4 set_mon_vals no callback set 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77981406a0 con 0x7f77980b5470 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.792+0000 7f779d8a4640 1 librados: init done 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.793+0000 7f779d8a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.793+0000 7f779d8a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.793+0000 7f779d8a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.793+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77980b5850 con 0x7f77980b5470 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.793+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f7790039040 con 0x7f77980b5470 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 1 --2- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 0x7f77780700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:21.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f77967fc640 1 --2- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 0x7f77780700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f77900ac1d0 con 0x7f77980b5470 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 10 client.26772.objecter ms_dispatch 0x7f7798000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 3 client.26772.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 3 client.26772.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f77967fc640 10 monclient: get_auth_request con 0x7f777806dc00 auth_method 0 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 20 client.26772.objecter dump_active .. 0 homeless 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f778ffff640 10 client.26772.objecter ms_handle_connect 0x7f777806dc00 2024-09-16T04:21:21.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.794+0000 7f77967fc640 1 --2- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 0x7f77780700c0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f7798140e80 tx=0x7f7780009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:21.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.798+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f779007a540 con 0x7f77980b5470 2024-09-16T04:21:21.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.798+0000 7f778ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:21.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.798+0000 7f778ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:21.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.977+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f7798072230 con 0x7f777806dc00 2024-09-16T04:21:21.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.978+0000 7f778ffff640 1 -- 172.21.15.70:0/410543504 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7798072230 con 0x7f777806dc00 2024-09-16T04:21:21.979 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 10 librados: watch_flush enter 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 10 librados: watch_flush exit 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 20 client.26772.objecter shutdown clearing up homeless session... 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 10 client.26772.objecter successfully canceled tick 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 msgr2=0x7f77780700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 0x7f77780700c0 secure :-1 s=READY pgs=4033 cs=0 l=1 rev1=1 crypto rx=0x7f7798140e80 tx=0x7f7780009040 comp rx=0 tx=0).stop 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 10 monclient: shutdown 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 msgr2=0x7f779813ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 secure :-1 s=READY pgs=5743 cs=0 l=1 rev1=1 crypto rx=0x7f7790000f80 tx=0x7f7790004a40 comp rx=0 tx=0).stop 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 shutdown_connections 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> v2:172.21.15.70:6804/198622594 conn(0x7f777806dc00 0x7f77780700c0 unknown :-1 s=CLOSED pgs=4033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f779813d020 0x7f7798133170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f779813cbd0 0x7f7798132c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 --2- 172.21.15.70:0/410543504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77980b5470 0x7f779813ff80 unknown :-1 s=CLOSED pgs=5743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 >> 172.21.15.70:0/410543504 conn(0x7f77980679c0 msgr2=0x7f7798138390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 shutdown_connections 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 -- 172.21.15.70:0/410543504 wait complete. 2024-09-16T04:21:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:21.980+0000 7f779d8a4640 1 librados: shutdown 2024-09-16T04:21:21.992 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:21:21.993 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 10 monclient: get_monmap_and_config 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 10 monclient: monmap: 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:22.094675+0000 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:22.094675+0000 2024-09-16T04:21:22.095 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:22.096 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 1 Processor -- start 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 1 -- start start 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.094+0000 7fbdd8f6d640 10 monclient: init 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdd40b5e80 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7fbdd40aace0 0x7fbdd4140380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd40aace0 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd414da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd41408c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd40ac070 con 0x7fbdd40b5e80 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7fbdd40b6260 con 0x7fbdd40aace0 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd40ac2e0 con 0x7fbdd41408c0 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd8f6d640 10 monclient(hunting): authenticate will time out at 2640.756653s 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbdd40b5e80 0x7fbdd41466e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42478/0 (socket says 172.21.15.70:42478) 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd259c640 1 -- 172.21.15.70:0/2886067318 learned_addr learned my addr 172.21.15.70:0/2886067318 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): get_auth_request con 0x7fbdd41408c0 auth_method 0 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.095+0000 7fbdd1d9b640 1 --2- 172.21.15.70:0/2886067318 >> v2:172.21.15.163:3300/0 conn(0x7fbdd40aace0 0x7fbdd4140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd1d9b640 10 monclient(hunting): get_auth_request con 0x7fbdd40aace0 auth_method 0 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_done global_id 20990 payload 274 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 1 -- 172.21.15.70:0/2886067318 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109447927 0 0) 0x7fbdd40ac070 con 0x7fbdd40b5e80 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 -- 172.21.15.70:0/2886067318 >> v2:172.21.15.163:3300/0 conn(0x7fbdd40aace0 msgr2=0x7fbdd4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 --2- 172.21.15.70:0/2886067318 >> v2:172.21.15.163:3300/0 conn(0x7fbdd40aace0 0x7fbdd4140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 -- 172.21.15.70:0/2886067318 >> v1:172.21.15.70:6789/0 conn(0x7fbdd40b5e80 legacy=0x7fbdd41466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 monclient: found mon.noname-b 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 -- 172.21.15.70:0/2886067318 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd40aa4e0 con 0x7fbdd41408c0 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd2d9d640 1 --2- 172.21.15.70:0/2886067318 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd414da10 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009d90 tx=0x7fbdcc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a774fbf544d7fa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 1 -- 172.21.15.70:0/2886067318 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc007740 con 0x7fbdd41408c0 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: dump: 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.108 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 1 -- 172.21.15.70:0/2886067318 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0078e0 con 0x7fbdd41408c0 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd8f6d640 5 monclient: authenticate success, global_id 20990 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd8f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd8f6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 1 -- 172.21.15.70:0/2886067318 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc007cb0 con 0x7fbdd41408c0 2024-09-16T04:21:22.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd8f6d640 4 set_mon_vals no callback set 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.096+0000 7fbdd159a640 10 monclient: dump: 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.110 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 10 monclient: shutdown 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2886067318 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 msgr2=0x7fbdd414da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2886067318 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd414da10 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009d90 tx=0x7fbdcc02a660 comp rx=0 tx=0).stop 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2886067318 shutdown_connections 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2886067318 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd414da10 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2886067318 >> v2:172.21.15.163:3300/0 conn(0x7fbdd40aace0 0x7fbdd4140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2886067318 >> 172.21.15.70:0/2886067318 conn(0x7fbdd4067ba0 msgr2=0x7fbdd4067fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2886067318 shutdown_connections 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.097+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2886067318 wait complete. 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 10 monclient: monmap: 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.111 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 librados: starting msgr at 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 librados: starting objecter 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 Processor -- start 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 -- start start 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 librados: setting wanted keys 2024-09-16T04:21:22.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 1 librados: calling monclient init 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.098+0000 7fbdd8f6d640 10 monclient: init 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 0x7fbdd40ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd40aace0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd40b5e80 0x7fbdd41d8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd40b5e80 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd41d8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd41408c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4146df0 con 0x7fbdd40aace0 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd4140d50 con 0x7fbdd40b5e80 2024-09-16T04:21:22.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd40b3350 con 0x7fbdd41408c0 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd8f6d640 10 monclient(hunting): authenticate will time out at 2640.760294s 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 0x7fbdd40ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd41d8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 0x7fbdd40ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33388/0 (socket says 172.21.15.70:33388) 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 -- 172.21.15.70:0/2968438469 learned_addr learned my addr 172.21.15.70:0/2968438469 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): get_auth_request con 0x7fbdd40aace0 auth_method 0 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): get_auth_request con 0x7fbdd41408c0 auth_method 0 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd40b5e80 0x7fbdd41d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): get_auth_request con 0x7fbdd40b5e80 auth_method 0 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient(hunting): handle_auth_done global_id 26778 payload 824 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 -- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 msgr2=0x7fbdd41d8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd41d8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 -- 172.21.15.70:0/2968438469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd40b5e80 msgr2=0x7fbdd41d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbdd40b5e80 0x7fbdd41d8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 monclient: found mon.noname-a 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 -- 172.21.15.70:0/2968438469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd40aebd0 con 0x7fbdd40aace0 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd2d9d640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fbdd41408c0 0x7fbdd41d8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdd259c640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 0x7fbdd40ad610 secure :-1 s=READY pgs=5745 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c4d0 tx=0x7fbdc400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 client.?.objecter ms_handle_connect 0x7fbdd40aace0 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4018070 con 0x7fbdd40aace0 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 monclient: dump: 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.117 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.099+0000 7fbdc2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc4007500 con 0x7fbdd40aace0 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 5 monclient: authenticate success, global_id 26778 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 monclient: _renew_subs 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4004a20 con 0x7fbdd40aace0 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd40aee40 con 0x7fbdd40aace0 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdc2ffd640 10 monclient: dump: 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.119 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd0d99640 4 set_mon_vals no callback set 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 client.26778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 monclient: _renew_subs 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd41d8c90 con 0x7fbdd40aace0 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 1 librados: init done 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 client.26778.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 15 client.26778.objecter _session_command_op_assign -1 1 2024-09-16T04:21:22.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 20 client.26778.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 client.26778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.100+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbdd40b54f0 con 0x7fbdd40aace0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.101+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fbdc401b070 con 0x7fbdd40aace0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.101+0000 7fbdc2ffd640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6804/198622594 conn(0x7fbdb406dc30 0x7fbdb40700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd1d9b640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6804/198622594 conn(0x7fbdb406dc30 0x7fbdb40700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7fbdc408a440 con 0x7fbdd40aace0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd1d9b640 10 monclient: get_auth_request con 0x7fbdb406dc30 auth_method 0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 client.26778.objecter ms_dispatch 0x7fbdd4000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 3 client.26778.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 3 client.26778.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 client.26778.objecter checking command 1 2024-09-16T04:21:22.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbdb4075550 0x7fbdb40779f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter _get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 15 client.26778.objecter _session_command_op_remove -1 1 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter _get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 15 client.26778.objecter _session_command_op_assign 3 1 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 4 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 client.26778.objecter _send_command 1 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 --> v2:172.21.15.70:6805/3986044637 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdb4078080 con 0x7fbdb4075550 2024-09-16T04:21:22.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 20 client.26778.objecter dump_active .. 0 homeless 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7fbdc4091050 con 0x7fbdd40aace0 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd2d9d640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbdb4075550 0x7fbdb40779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd2d9d640 10 monclient: get_auth_request con 0x7fbdb4075550 auth_method 0 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd1d9b640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6804/198622594 conn(0x7fbdb406dc30 0x7fbdb40700f0 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7fbdd41468d0 tx=0x7fbdc8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 client.26778.objecter ms_handle_connect 0x7fbdb406dc30 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdc2ffd640 10 client.26778.objecter ms_handle_connect 0x7fbdb4075550 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.102+0000 7fbdd2d9d640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbdb4075550 0x7fbdb40779f0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.103+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== osd.3 v2:172.21.15.70:6805/3986044637 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbdb4078080 con 0x7fbdb4075550 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.103+0000 7fbdc2ffd640 10 client.26778.objecter ms_dispatch 0x7fbdd4000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:22.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.103+0000 7fbdc2ffd640 10 client.26778.objecter _finish_command 1 = system:0 2024-09-16T04:21:22.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.103+0000 7fbdc2ffd640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 4 2024-09-16T04:21:22.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.103+0000 7fbdc2ffd640 15 client.26778.objecter _session_command_op_remove 3 1 2024-09-16T04:21:22.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 10 client.26778.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:22.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 15 client.26778.objecter _session_command_op_assign -1 2 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 20 client.26778.objecter _get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 20 client.26778.objecter _get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 15 client.26778.objecter _session_command_op_remove -1 2 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 20 client.26778.objecter get_session s=0x7fbdb4074ea0 osd=3 3 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 15 client.26778.objecter _session_command_op_assign 3 2 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 4 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 10 client.26778.objecter _send_command 2 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 --> v2:172.21.15.70:6805/3986044637 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbdd40ae3a0 con 0x7fbdb4075550 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdc2ffd640 1 -- 172.21.15.70:0/2968438469 <== osd.3 v2:172.21.15.70:6805/3986044637 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbdd40ae3a0 con 0x7fbdb4075550 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdc2ffd640 10 client.26778.objecter ms_dispatch 0x7fbdd4000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdc2ffd640 10 client.26778.objecter _finish_command 2 = system:0 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdc2ffd640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 4 2024-09-16T04:21:22.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.115+0000 7fbdc2ffd640 15 client.26778.objecter _session_command_op_remove 3 2 2024-09-16T04:21:22.126 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:22.126 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:22.126 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:22.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 10 librados: watch_flush enter 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 10 librados: watch_flush exit 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 10 client.26778.objecter close_session for osd.3 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbdb4075550 msgr2=0x7fbdb40779f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6805/3986044637 conn(0x7fbdb4075550 0x7fbdb40779f0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 20 client.26778.objecter put_session s=0x7fbdb4074ea0 osd=3 1 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 20 client.26778.objecter shutdown clearing up homeless session... 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 10 client.26778.objecter successfully canceled tick 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6804/198622594 conn(0x7fbdb406dc30 msgr2=0x7fbdb40700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2968438469 >> v2:172.21.15.70:6804/198622594 conn(0x7fbdb406dc30 0x7fbdb40700f0 secure :-1 s=READY pgs=4034 cs=0 l=1 rev1=1 crypto rx=0x7fbdd41468d0 tx=0x7fbdc8008040 comp rx=0 tx=0).stop 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 10 monclient: shutdown 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 msgr2=0x7fbdd40ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 --2- 172.21.15.70:0/2968438469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdd40aace0 0x7fbdd40ad610 secure :-1 s=READY pgs=5745 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c4d0 tx=0x7fbdc400c990 comp rx=0 tx=0).stop 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd2d9d640 1 -- 172.21.15.70:0/2968438469 reap_dead start 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 shutdown_connections 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 >> 172.21.15.70:0/2968438469 conn(0x7fbdd4067ba0 msgr2=0x7fbdd40b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 shutdown_connections 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 -- 172.21.15.70:0/2968438469 wait complete. 2024-09-16T04:21:22.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.116+0000 7fbdd8f6d640 1 librados: shutdown 2024-09-16T04:21:22.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:21:22.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.226+0000 7fa95de5e640 10 monclient: get_monmap_and_config 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.226+0000 7fa95de5e640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.226+0000 7fa95de5e640 10 monclient: monmap: 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:22.227051+0000 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:22.227051+0000 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.228 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:22.229 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:22.229 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:22.229 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:22.229 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:22.229 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 Processor -- start 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 -- start start 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient: init 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9580b37a0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:22.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa95813cb40 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa95813cf90 0x7fa958143800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-a con 0x7fa95813cf90 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9580abcf0 con 0x7fa9580b37a0 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9580abf60 con 0x7fa95813cb40 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa9580ac240 con 0x7fa95813cf90 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.227+0000 7fa95de5e640 10 monclient(hunting): authenticate will time out at 2640.888968s 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa95813cf90 0x7fa958143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa9577fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa9580b37a0 0x7fa9580b3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42492/0 (socket says 172.21.15.70:42492) 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa9577fe640 1 -- 172.21.15.70:0/2521148324 learned_addr learned my addr 172.21.15.70:0/2521148324 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): get_auth_request con 0x7fa95813cf90 auth_method 0 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient(hunting): handle_auth_done global_id 26781 payload 274 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 1 -- 172.21.15.70:0/2521148324 >> v1:172.21.15.163:6789/0 conn(0x7fa95813cb40 legacy=0x7fa9581430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 1 -- 172.21.15.70:0/2521148324 >> v1:172.21.15.70:6789/0 conn(0x7fa9580b37a0 legacy=0x7fa9580b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 1 monclient: found mon.noname-a 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.228+0000 7fa957fff640 1 -- 172.21.15.70:0/2521148324 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9580b4590 con 0x7fa95813cf90 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa957fff640 1 --2- 172.21.15.70:0/2521148324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa95813cf90 0x7fa958143800 secure :-1 s=READY pgs=5747 cs=0 l=1 rev1=1 crypto rx=0x7fa950009d90 tx=0x7fa95002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f8aefa29873f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 1 -- 172.21.15.70:0/2521148324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950031600 con 0x7fa95813cf90 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: dump: 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.240 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 1 -- 172.21.15.70:0/2521148324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9500317a0 con 0x7fa95813cf90 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 5 monclient: authenticate success, global_id 26781 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 1 -- 172.21.15.70:0/2521148324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950031b70 con 0x7fa95813cf90 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 4 set_mon_vals no callback set 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa9567fc640 10 monclient: dump: 2024-09-16T04:21:22.241 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 10 monclient: shutdown 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 -- 172.21.15.70:0/2521148324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa95813cf90 msgr2=0x7fa958143800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 --2- 172.21.15.70:0/2521148324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa95813cf90 0x7fa958143800 secure :-1 s=READY pgs=5747 cs=0 l=1 rev1=1 crypto rx=0x7fa950009d90 tx=0x7fa95002a5d0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 -- 172.21.15.70:0/2521148324 shutdown_connections 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 --2- 172.21.15.70:0/2521148324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa95813cf90 0x7fa958143800 unknown :-1 s=CLOSED pgs=5747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 -- 172.21.15.70:0/2521148324 >> 172.21.15.70:0/2521148324 conn(0x7fa9580679c0 msgr2=0x7fa958067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 -- 172.21.15.70:0/2521148324 shutdown_connections 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.229+0000 7fa95de5e640 1 -- 172.21.15.70:0/2521148324 wait complete. 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 10 monclient: monmap: 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 1 librados: starting msgr at 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 1 librados: starting objecter 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 1 Processor -- start 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.230+0000 7fa95de5e640 1 -- start start 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 librados: setting wanted keys 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 librados: calling monclient init 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient: init 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9580b37a0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 0x7fa9580b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-c con 0x7fa95813cb40 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 0x7fa9580ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): picked mon.noname-b con 0x7fa95813cf90 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa95814e7f0 con 0x7fa9580b37a0 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa958143d40 con 0x7fa95813cf90 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7fa95814e970 con 0x7fa95813cb40 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa95de5e640 10 monclient(hunting): authenticate will time out at 2640.892542s 2024-09-16T04:21:22.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33400/0 (socket says 172.21.15.70:33400) 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 0x7fa9580b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 -- 172.21.15.70:0/265978926 learned_addr learned my addr 172.21.15.70:0/265978926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): get_auth_request con 0x7fa9580b37a0 auth_method 0 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): get_auth_request con 0x7fa95813cb40 auth_method 0 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 0x7fa9580ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 10 monclient(hunting): get_auth_request con 0x7fa95813cf90 auth_method 0 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient(hunting): handle_auth_done global_id 26787 payload 824 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 -- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 msgr2=0x7fa9580b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 0x7fa9580b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 -- 172.21.15.70:0/265978926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 msgr2=0x7fa9580ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 0x7fa9580ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 monclient: found mon.noname-a 2024-09-16T04:21:22.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa9577fe640 1 -- 172.21.15.70:0/265978926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9580b0420 con 0x7fa9580b37a0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa957fff640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 0x7fa9580ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.231+0000 7fa956ffd640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 0x7fa9580b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa9577fe640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 secure :-1 s=READY pgs=5748 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c4a0 tx=0x7fa94c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 client.?.objecter ms_handle_connect 0x7fa9580b37a0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c017070 con 0x7fa9580b37a0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: dump: 2024-09-16T04:21:22.248 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 5 monclient: authenticate success, global_id 26787 2024-09-16T04:21:22.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa94c005ce0 con 0x7fa9580b37a0 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 10 monclient: _renew_subs 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94c007270 con 0x7fa9580b37a0 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9580ae570 con 0x7fa9580b37a0 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa93ffff640 10 monclient: dump: 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.250 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa955ffb640 4 set_mon_vals no callback set 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 10 client.26787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 10 monclient: _renew_subs 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9580af100 con 0x7fa9580b37a0 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.232+0000 7fa95de5e640 1 librados: init done 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.233+0000 7fa95de5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.233+0000 7fa95de5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.233+0000 7fa95de5e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.233+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9580b3b80 con 0x7fa9580b37a0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.233+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7fa94c01a070 con 0x7fa9580b37a0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 1 --2- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 0x7fa9440700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa956ffd640 1 --2- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 0x7fa9440700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7fa94c08a500 con 0x7fa9580b37a0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 10 client.26787.objecter ms_dispatch 0x7fa958000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 3 client.26787.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 3 client.26787.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa956ffd640 10 monclient: get_auth_request con 0x7fa94406dc00 auth_method 0 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 20 client.26787.objecter dump_active .. 0 homeless 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa93ffff640 10 client.26787.objecter ms_handle_connect 0x7fa94406dc00 2024-09-16T04:21:22.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.234+0000 7fa956ffd640 1 --2- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 0x7fa9440700c0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7fa9580b57b0 tx=0x7fa948009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.238+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa94c0587e0 con 0x7fa9580b37a0 2024-09-16T04:21:22.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.238+0000 7fa93ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.238+0000 7fa93ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:22.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.407+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fa958140e70 con 0x7fa94406dc00 2024-09-16T04:21:22.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.407+0000 7fa93ffff640 1 -- 172.21.15.70:0/265978926 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa958140e70 con 0x7fa94406dc00 2024-09-16T04:21:22.409 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:21:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 10 librados: watch_flush enter 2024-09-16T04:21:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 10 librados: watch_flush exit 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 20 client.26787.objecter shutdown clearing up homeless session... 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 10 client.26787.objecter successfully canceled tick 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 msgr2=0x7fa9440700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 0x7fa9440700c0 secure :-1 s=READY pgs=4035 cs=0 l=1 rev1=1 crypto rx=0x7fa9580b57b0 tx=0x7fa948009040 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 10 monclient: shutdown 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 msgr2=0x7fa958146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 secure :-1 s=READY pgs=5748 cs=0 l=1 rev1=1 crypto rx=0x7fa94c00c4a0 tx=0x7fa94c00c960 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 shutdown_connections 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> v2:172.21.15.70:6804/198622594 conn(0x7fa94406dc00 0x7fa9440700c0 unknown :-1 s=CLOSED pgs=4035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa95813cf90 0x7fa9580ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7fa95813cb40 0x7fa9580b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 --2- 172.21.15.70:0/265978926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9580b37a0 0x7fa958146910 unknown :-1 s=CLOSED pgs=5748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.409+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 >> 172.21.15.70:0/265978926 conn(0x7fa9580679c0 msgr2=0x7fa958141390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.410+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 shutdown_connections 2024-09-16T04:21:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.410+0000 7fa95de5e640 1 -- 172.21.15.70:0/265978926 wait complete. 2024-09-16T04:21:22.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.410+0000 7fa95de5e640 1 librados: shutdown 2024-09-16T04:21:22.421 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:21:22.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:22.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.521+0000 7f1622c23640 10 monclient: get_monmap_and_config 2024-09-16T04:21:22.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.521+0000 7f1622c23640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.521+0000 7f1622c23640 10 monclient: monmap: 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:22.521829+0000 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:22.521829+0000 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:22.523 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:22.524 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:22.524 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 Processor -- start 2024-09-16T04:21:22.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 -- start start 2024-09-16T04:21:22.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient: init 2024-09-16T04:21:22.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f161c06f740 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f161c070890 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f161c06fb20 0x7f161c14a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-c con 0x7f161c06fb20 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c071a00 con 0x7f161c070890 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c070440 con 0x7f161c06f740 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f161c0b0410 con 0x7f161c06fb20 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1622c23640 10 monclient(hunting): authenticate will time out at 2641.183794s 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1613fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f161c070890 0x7f161c0b0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42500/0 (socket says 172.21.15.70:42500) 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.522+0000 7f1613fff640 1 -- 172.21.15.70:0/1071979977 learned_addr learned my addr 172.21.15.70:0/1071979977 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 1 --2- 172.21.15.70:0/1071979977 >> v2:172.21.15.163:3300/0 conn(0x7f161c06fb20 0x7f161c14a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): get_auth_request con 0x7f161c06fb20 auth_method 0 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039526213 0 0) 0x7f161c071a00 con 0x7f161c070890 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): _negotiate 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): my global_id is 26790 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1604004630 con 0x7f161c070890 2024-09-16T04:21:22.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 50878071 0 0) 0x7f1604004630 con 0x7f161c070890 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 >> v2:172.21.15.163:3300/0 conn(0x7f161c06fb20 msgr2=0x7f161c14a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 --2- 172.21.15.70:0/1071979977 >> v2:172.21.15.163:3300/0 conn(0x7f161c06fb20 0x7f161c14a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 >> v1:172.21.15.163:6789/0 conn(0x7f161c06f740 legacy=0x7f161c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T04:21:22.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c14c5a0 con 0x7f161c070890 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f160c001670 con 0x7f161c070890 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: dump: 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.541 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 >> v1:172.21.15.70:6789/0 conn(0x7f161c070890 legacy=0x7f161c0b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 0x7f1604004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): picked mon.a con 0x7f1604004ac0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1604005910 0x7f1604005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): picked mon.b con 0x7f1604005910 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f16040062d0 0x7f160400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): picked mon.c con 0x7f16040062d0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f161c0b0410 con 0x7f1604004ac0 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f160400d170 con 0x7f1604005910 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f160400d370 con 0x7f16040062d0 2024-09-16T04:21:22.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16137fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 0x7f1604004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): get_auth_request con 0x7f1604004ac0 auth_method 0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1604005910 0x7f1604005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f16040062d0 0x7f160400cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): get_auth_request con 0x7f16040062d0 auth_method 0 2024-09-16T04:21:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): get_auth_request con 0x7f1604005910 auth_method 0 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient(hunting): handle_auth_done global_id 26790 payload 293 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 -- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f16040062d0 msgr2=0x7f160400cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f16040062d0 0x7f160400cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 -- 172.21.15.70:0/1071979977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1604005910 msgr2=0x7f1604005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1604005910 0x7f1604005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 monclient: found mon.a 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 1 -- 172.21.15.70:0/1071979977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1604004630 con 0x7f1604004ac0 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16209bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f16211c0640 1 -- 172.21.15.70:0/1071979977 reap_dead start 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1622c23640 5 monclient: authenticate success, global_id 26790 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.523+0000 7f1622c23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16209bf640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 0x7f1604004ee0 secure :-1 s=READY pgs=5750 cs=0 l=1 rev1=1 crypto rx=0x7f1618001750 tx=0x7f1604006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c455443064c148 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618001370 con 0x7f1604004ac0 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: dump: 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.548 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16180052d0 con 0x7f1604004ac0 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f1622c23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f1622c23640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 1 -- 172.21.15.70:0/1071979977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161802abc0 con 0x7f1604004ac0 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f1622c23640 4 set_mon_vals no callback set 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.524+0000 7f16137fe640 10 monclient: dump: 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.549 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 10 monclient: shutdown 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 -- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 msgr2=0x7f1604004ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 0x7f1604004ee0 secure :-1 s=READY pgs=5750 cs=0 l=1 rev1=1 crypto rx=0x7f1618001750 tx=0x7f1604006940 comp rx=0 tx=0).stop 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 -- 172.21.15.70:0/1071979977 shutdown_connections 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 --2- 172.21.15.70:0/1071979977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1604004ac0 0x7f1604004ee0 unknown :-1 s=CLOSED pgs=5750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 -- 172.21.15.70:0/1071979977 >> 172.21.15.70:0/1071979977 conn(0x7f161c0b4be0 msgr2=0x7f161c0b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 -- 172.21.15.70:0/1071979977 shutdown_connections 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 1 -- 172.21.15.70:0/1071979977 wait complete. 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.525+0000 7f1622c23640 10 monclient: monmap: 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:22.551 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 librados: starting msgr at 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 librados: starting objecter 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 Processor -- start 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 -- start start 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 librados: setting wanted keys 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 librados: calling monclient init 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient: init 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 0x7f161c13eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-c con 0x7f1604004ac0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f161c141880 0x7f161c1d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-a con 0x7f161c141880 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f161c142260 0x7f161c13f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): picked mon.noname-b con 0x7f161c142260 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f161c0af970 con 0x7f161c141880 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f161c0af7f0 con 0x7f161c142260 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f161c0afaf0 con 0x7f1604004ac0 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f1622c23640 10 monclient(hunting): authenticate will time out at 2641.187931s 2024-09-16T04:21:22.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.526+0000 7f16209bf640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 0x7f161c13eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f161c141880 0x7f161c1d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 0x7f161c13eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54864/0 (socket says 172.21.15.70:54864) 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 -- 172.21.15.70:0/707026353 learned_addr learned my addr 172.21.15.70:0/707026353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): get_auth_request con 0x7f1604004ac0 auth_method 0 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): get_auth_request con 0x7f161c141880 auth_method 0 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16211c0640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f161c142260 0x7f161c13f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16211c0640 10 monclient(hunting): get_auth_request con 0x7f161c142260 auth_method 0 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16211c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16211c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16211c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient(hunting): handle_auth_done global_id 20999 payload 824 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 -- 172.21.15.70:0/707026353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f161c142260 msgr2=0x7f161c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f161c142260 0x7f161c13f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 -- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f161c141880 msgr2=0x7f161c1d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f161c141880 0x7f161c1d98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 monclient: found mon.noname-c 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 -- 172.21.15.70:0/707026353 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c143130 con 0x7f1604004ac0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1613fff640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f161c141880 0x7f161c1d98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16209bf640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 0x7f161c13eb60 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f1618001750 tx=0x7f1618001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 client.?.objecter ms_handle_connect 0x7f1604004ac0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161803f020 con 0x7f1604004ac0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: dump: 2024-09-16T04:21:22.556 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1618005480 con 0x7f1604004ac0 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 5 monclient: authenticate success, global_id 20999 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 10 monclient: _renew_subs 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c143410 con 0x7f1604004ac0 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618037040 con 0x7f1604004ac0 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f16117fa640 10 monclient: dump: 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.558 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 10 client.20999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 10 monclient: _renew_subs 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1612ffd640 4 set_mon_vals no callback set 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c1426f0 con 0x7f1604004ac0 2024-09-16T04:21:22.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.527+0000 7f1622c23640 1 librados: init done 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 10 client.20999.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 15 client.20999.objecter _session_command_op_assign -1 1 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 20 client.20999.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 10 client.20999.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.528+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f161c072a00 con 0x7f1604004ac0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f1618042020 con 0x7f1604004ac0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.70:6804/198622594 conn(0x7f160806dc00 0x7f16080700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f1613fff640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.70:6804/198622594 conn(0x7f160806dc00 0x7f16080700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f16180aaca0 con 0x7f1604004ac0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 10 client.20999.objecter ms_dispatch 0x7f161c000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 3 client.20999.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 3 client.20999.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:22.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f1613fff640 10 monclient: get_auth_request con 0x7f160806dc00 auth_method 0 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 10 client.20999.objecter checking command 1 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.163:6804/1904387775 conn(0x7f1608075520 0x7f16080779c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter _get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 15 client.20999.objecter _session_command_op_remove -1 1 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter _get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 15 client.20999.objecter _session_command_op_assign 4 1 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 4 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 10 client.20999.objecter _send_command 1 2024-09-16T04:21:22.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 --> v2:172.21.15.163:6804/1904387775 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16180052d0 con 0x7f1608075520 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 20 client.20999.objecter dump_active .. 0 homeless 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7f16180ad3a0 con 0x7f1604004ac0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.529+0000 7f16117fa640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f16211c0640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.163:6804/1904387775 conn(0x7f1608075520 0x7f16080779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f1613fff640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.70:6804/198622594 conn(0x7f160806dc00 0x7f16080700c0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7f160c0017e0 tx=0x7f160c000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f16117fa640 10 client.20999.objecter ms_handle_connect 0x7f160806dc00 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f16211c0640 10 monclient: get_auth_request con 0x7f1608075520 auth_method 0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f16117fa640 10 client.20999.objecter ms_handle_connect 0x7f1608075520 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.530+0000 7f16211c0640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.163:6804/1904387775 conn(0x7f1608075520 0x7f16080779c0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.531+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== osd.4 v2:172.21.15.163:6804/1904387775 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16180052d0 con 0x7f1608075520 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.531+0000 7f16117fa640 10 client.20999.objecter ms_dispatch 0x7f161c000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.531+0000 7f16117fa640 10 client.20999.objecter _finish_command 1 = system:0 2024-09-16T04:21:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.531+0000 7f16117fa640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 4 2024-09-16T04:21:22.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.531+0000 7f16117fa640 15 client.20999.objecter _session_command_op_remove 4 1 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 10 client.20999.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 15 client.20999.objecter _session_command_op_assign -1 2 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 20 client.20999.objecter _get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 20 client.20999.objecter _get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 15 client.20999.objecter _session_command_op_remove -1 2 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 20 client.20999.objecter get_session s=0x7f1608074e70 osd=4 3 2024-09-16T04:21:22.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 15 client.20999.objecter _session_command_op_assign 4 2 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 4 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 10 client.20999.objecter _send_command 2 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 --> v2:172.21.15.163:6804/1904387775 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f161c0b08a0 con 0x7f1608075520 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f16117fa640 1 -- 172.21.15.70:0/707026353 <== osd.4 v2:172.21.15.163:6804/1904387775 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f161c0b08a0 con 0x7f1608075520 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f16117fa640 10 client.20999.objecter ms_dispatch 0x7f161c000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f16117fa640 10 client.20999.objecter _finish_command 2 = system:0 2024-09-16T04:21:22.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f16117fa640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 4 2024-09-16T04:21:22.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.543+0000 7f16117fa640 15 client.20999.objecter _session_command_op_remove 4 2 2024-09-16T04:21:22.566 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:22.566 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:22.566 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 10 librados: watch_flush enter 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 10 librados: watch_flush exit 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 10 client.20999.objecter close_session for osd.4 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 >> v2:172.21.15.163:6804/1904387775 conn(0x7f1608075520 msgr2=0x7f16080779c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.163:6804/1904387775 conn(0x7f1608075520 0x7f16080779c0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 20 client.20999.objecter put_session s=0x7f1608074e70 osd=4 1 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 20 client.20999.objecter shutdown clearing up homeless session... 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 10 client.20999.objecter successfully canceled tick 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 >> v2:172.21.15.70:6804/198622594 conn(0x7f160806dc00 msgr2=0x7f16080700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 --2- 172.21.15.70:0/707026353 >> v2:172.21.15.70:6804/198622594 conn(0x7f160806dc00 0x7f16080700c0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7f160c0017e0 tx=0x7f160c000ed0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 10 monclient: shutdown 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 msgr2=0x7f161c13eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 --2- 172.21.15.70:0/707026353 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f1604004ac0 0x7f161c13eb60 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f1618001750 tx=0x7f1618001120 comp rx=0 tx=0).stop 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f16211c0640 1 -- 172.21.15.70:0/707026353 reap_dead start 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 shutdown_connections 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 >> 172.21.15.70:0/707026353 conn(0x7f161c0b4be0 msgr2=0x7f161c0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 shutdown_connections 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 -- 172.21.15.70:0/707026353 wait complete. 2024-09-16T04:21:22.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.544+0000 7f1622c23640 1 librados: shutdown 2024-09-16T04:21:22.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 10 monclient: get_monmap_and_config 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 10 monclient: monmap: 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:22.668639+0000 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:22.668639+0000 2024-09-16T04:21:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:22.670 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 1 Processor -- start 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 1 -- start start 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.668+0000 7f9476c17640 10 monclient: init 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f94700b5650 0x7f94700b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-c con 0x7f94700b5650 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9470140410 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9470140860 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94700aae40 con 0x7f9470140860 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94700b5cd0 con 0x7f9470140410 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f94700b5a30 con 0x7f94700b5650 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f9476c17640 10 monclient(hunting): authenticate will time out at 2641.330558s 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f94751b4640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9470140860 0x7f9470146c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42514/0 (socket says 172.21.15.70:42514) 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f94751b4640 1 -- 172.21.15.70:0/1553616136 learned_addr learned my addr 172.21.15.70:0/1553616136 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446460315 0 0) 0x7f94700aae40 con 0x7f9470140860 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 10 monclient(hunting): my global_id is 26796 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f945c003520 con 0x7f9470140860 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.669+0000 7f94749b3640 1 --2- 172.21.15.70:0/1553616136 >> v2:172.21.15.163:3300/0 conn(0x7f94700b5650 0x7f94700b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): get_auth_request con 0x7f94700b5650 auth_method 0 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3426784994 0 0) 0x7f945c003520 con 0x7f9470140860 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 >> v2:172.21.15.163:3300/0 conn(0x7f94700b5650 msgr2=0x7f94700b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 --2- 172.21.15.70:0/1553616136 >> v2:172.21.15.163:3300/0 conn(0x7f94700b5650 0x7f94700b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 >> v1:172.21.15.163:6789/0 conn(0x7f9470140410 legacy=0x7f9470146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94700aa9f0 con 0x7f9470140860 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f9476c17640 5 monclient: authenticate success, global_id 26796 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f9468001670 con 0x7f9470140860 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f9476c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: dump: 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.685 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 >> v1:172.21.15.70:6789/0 conn(0x7f9470140860 legacy=0x7f9470146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f945c0039b0 0x7f945c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): picked mon.b con 0x7f945c0039b0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 0x7f945c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): picked mon.a con 0x7f945c004800 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c0051c0 0x7f945c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): picked mon.c con 0x7f945c0051c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94700b5a30 con 0x7f945c004800 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f945c00c160 con 0x7f945c0039b0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f945c00c360 con 0x7f945c0051c0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f9476c17640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 0x7f945c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f945c0039b0 0x7f945c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c0051c0 0x7f945c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): get_auth_request con 0x7f945c0039b0 auth_method 0 2024-09-16T04:21:22.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): get_auth_request con 0x7f945c004800 auth_method 0 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): get_auth_request con 0x7f945c0051c0 auth_method 0 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient(hunting): handle_auth_done global_id 26796 payload 293 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 -- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c0051c0 msgr2=0x7f945c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c0051c0 0x7f945c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 -- 172.21.15.70:0/1553616136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f945c0039b0 msgr2=0x7f945c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f945c0039b0 0x7f945c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 monclient: found mon.a 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 1 -- 172.21.15.70:0/1553616136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c003520 con 0x7f945c004800 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f946ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c0051c0 0x7f945c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94749b3640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f945c0039b0 0x7f945c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.670+0000 7f94751b4640 1 -- 172.21.15.70:0/1553616136 reap_dead start 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946ffff640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 0x7f945c004c80 secure :-1 s=READY pgs=5752 cs=0 l=1 rev1=1 crypto rx=0x7f9458001750 tx=0x7f945c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0723bb6d5a0585 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458001370 con 0x7f945c004800 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: dump: 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94580052d0 con 0x7f945c004800 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f9476c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f9476c17640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 1 -- 172.21.15.70:0/1553616136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945802abc0 con 0x7f945c004800 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f9476c17640 4 set_mon_vals no callback set 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.671+0000 7f946f7fe640 10 monclient: dump: 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 10 monclient: shutdown 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 -- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 msgr2=0x7f945c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 0x7f945c004c80 secure :-1 s=READY pgs=5752 cs=0 l=1 rev1=1 crypto rx=0x7f9458001750 tx=0x7f945c005830 comp rx=0 tx=0).stop 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 -- 172.21.15.70:0/1553616136 shutdown_connections 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 --2- 172.21.15.70:0/1553616136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f945c004800 0x7f945c004c80 unknown :-1 s=CLOSED pgs=5752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 -- 172.21.15.70:0/1553616136 >> 172.21.15.70:0/1553616136 conn(0x7f94700679c0 msgr2=0x7f9470067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 -- 172.21.15.70:0/1553616136 shutdown_connections 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 1 -- 172.21.15.70:0/1553616136 wait complete. 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.672+0000 7f9476c17640 10 monclient: monmap: 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 librados: starting msgr at 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 librados: starting objecter 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 Processor -- start 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 -- start start 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 librados: setting wanted keys 2024-09-16T04:21:22.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 librados: calling monclient init 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient: init 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-c con 0x7f945c004800 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-a con 0x7f94700b0c40 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f947013ce40 0x7f94701dbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): picked mon.noname-b con 0x7f947013ce40 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94700b3330 con 0x7f94700b0c40 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94700aac20 con 0x7f947013ce40 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f94700b34b0 con 0x7f945c004800 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.673+0000 7f9476c17640 10 monclient(hunting): authenticate will time out at 2641.334982s 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54888/0 (socket says 172.21.15.70:54888) 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33428/0 (socket says 172.21.15.70:33428) 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 -- 172.21.15.70:0/3598329460 learned_addr learned my addr 172.21.15.70:0/3598329460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): get_auth_request con 0x7f945c004800 auth_method 0 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): get_auth_request con 0x7f94700b0c40 auth_method 0 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient(hunting): handle_auth_done global_id 21002 payload 824 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 -- 172.21.15.70:0/3598329460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f947013ce40 msgr2=0x7f94701dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f947013ce40 0x7f94701dbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 -- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 msgr2=0x7f947013c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 monclient: found mon.noname-c 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 -- 172.21.15.70:0/3598329460 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9470120b20 con 0x7f945c004800 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946ffff640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f94749b3640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f946000f4c0 tx=0x7f946000f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 client.?.objecter ms_handle_connect 0x7f945c004800 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460015840 con 0x7f945c004800 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:22.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: dump: 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 5 monclient: authenticate success, global_id 21002 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94600159e0 con 0x7f945c004800 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 10 monclient: _renew_subs 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947013d2c0 con 0x7f945c004800 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460015db0 con 0x7f945c004800 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946d7fa640 10 monclient: dump: 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f946effd640 4 set_mon_vals no callback set 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 10 client.21002.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 10 monclient: _renew_subs 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94700b12a0 con 0x7f945c004800 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.674+0000 7f9476c17640 1 librados: init done 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.675+0000 7f9476c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.675+0000 7f9476c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.675+0000 7f9476c17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.675+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9470146500 con 0x7f945c004800 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.675+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f946001d400 con 0x7f945c004800 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 1 --2- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 0x7f94500700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946ffff640 1 --2- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 0x7f94500700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f946008c610 con 0x7f945c004800 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946ffff640 10 monclient: get_auth_request con 0x7f945006dc30 auth_method 0 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 10 client.21002.objecter ms_dispatch 0x7f9470000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 3 client.21002.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 3 client.21002.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 20 client.21002.objecter dump_active .. 0 homeless 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946d7fa640 10 client.21002.objecter ms_handle_connect 0x7f945006dc30 2024-09-16T04:21:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.676+0000 7f946ffff640 1 --2- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 0x7f94500700f0 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f9458032000 tx=0x7f945802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.680+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9460019030 con 0x7f945c004800 2024-09-16T04:21:22.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.680+0000 7f946d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.680+0000 7f946d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:22.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.847+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f94700b89a0 con 0x7f945006dc30 2024-09-16T04:21:22.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.848+0000 7f946d7fa640 1 -- 172.21.15.70:0/3598329460 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f94700b89a0 con 0x7f945006dc30 2024-09-16T04:21:22.849 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:21:22.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.849+0000 7f9476c17640 10 librados: watch_flush enter 2024-09-16T04:21:22.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.849+0000 7f9476c17640 10 librados: watch_flush exit 2024-09-16T04:21:22.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.849+0000 7f9476c17640 20 client.21002.objecter shutdown clearing up homeless session... 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.849+0000 7f9476c17640 10 client.21002.objecter successfully canceled tick 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 msgr2=0x7f94500700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 0x7f94500700f0 secure :-1 s=READY pgs=4037 cs=0 l=1 rev1=1 crypto rx=0x7f9458032000 tx=0x7f945802e000 comp rx=0 tx=0).stop 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 10 monclient: shutdown 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 msgr2=0x7f94700b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f946000f4c0 tx=0x7f946000f8c0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 shutdown_connections 2024-09-16T04:21:22.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> v2:172.21.15.70:6804/198622594 conn(0x7f945006dc30 0x7f94500700f0 unknown :-1 s=CLOSED pgs=4037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f947013ce40 0x7f94701dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94700b0c40 0x7f947013c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 --2- 172.21.15.70:0/3598329460 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f945c004800 0x7f94700b1500 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 >> 172.21.15.70:0/3598329460 conn(0x7f94700679c0 msgr2=0x7f9470144300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 shutdown_connections 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 -- 172.21.15.70:0/3598329460 wait complete. 2024-09-16T04:21:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.850+0000 7f9476c17640 1 librados: shutdown 2024-09-16T04:21:22.862 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:21:22.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.963+0000 7f788fc27640 10 monclient: get_monmap_and_config 2024-09-16T04:21:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.963+0000 7f788fc27640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.963+0000 7f788fc27640 10 monclient: monmap: 2024-09-16T04:21:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:22.963695+0000 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:22.963695+0000 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:22.965 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.963+0000 7f788fc27640 1 Processor -- start 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 -- start start 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient: init 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78880b5e80 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78880aace0 0x7f7888140380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-a con 0x7f78880aace0 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 0x7f788814da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-b con 0x7f78881408c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78880ac070 con 0x7f78880b5e80 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f78880b6260 con 0x7f78880aace0 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f78880ac2e0 con 0x7f78881408c0 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788fc27640 10 monclient(hunting): authenticate will time out at 2641.625656s 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 0x7f788814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f78880aace0 0x7f7888140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d9c3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f78880b5e80 0x7f78881466e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42528/0 (socket says 172.21.15.70:42528) 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d9c3640 1 -- 172.21.15.70:0/3792405480 learned_addr learned my addr 172.21.15.70:0/3792405480 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): get_auth_request con 0x7f78880aace0 auth_method 0 2024-09-16T04:21:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): get_auth_request con 0x7f78881408c0 auth_method 0 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.964+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 10 monclient(hunting): handle_auth_done global_id 26802 payload 274 2024-09-16T04:21:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 -- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 msgr2=0x7f788814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 0x7f788814da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 -- 172.21.15.70:0/3792405480 >> v1:172.21.15.70:6789/0 conn(0x7f78880b5e80 legacy=0x7f78881466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 monclient: found mon.noname-a 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 1 -- 172.21.15.70:0/3792405480 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823308925 0 0) 0x7f78880ac070 con 0x7f78880b5e80 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 -- 172.21.15.70:0/3792405480 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78880aa4e0 con 0x7f78880aace0 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788e1c4640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 0x7f788814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788d1c2640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78880aace0 0x7f7888140380 secure :-1 s=READY pgs=5754 cs=0 l=1 rev1=1 crypto rx=0x7f78780097a0 tx=0x7f787802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e271450d6b30ba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 1 -- 172.21.15.70:0/3792405480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878037070 con 0x7f78880aace0 2024-09-16T04:21:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: dump: 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788fc27640 5 monclient: authenticate success, global_id 26802 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788fc27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 1 -- 172.21.15.70:0/3792405480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7878032440 con 0x7f78880aace0 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788fc27640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 1 -- 172.21.15.70:0/3792405480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878032810 con 0x7f78880aace0 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788fc27640 4 set_mon_vals no callback set 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.965+0000 7f788c9c1640 10 monclient: dump: 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 10 monclient: shutdown 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 -- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78880aace0 msgr2=0x7f7888140380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78880aace0 0x7f7888140380 secure :-1 s=READY pgs=5754 cs=0 l=1 rev1=1 crypto rx=0x7f78780097a0 tx=0x7f787802a180 comp rx=0 tx=0).stop 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 -- 172.21.15.70:0/3792405480 shutdown_connections 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78881408c0 0x7f788814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 --2- 172.21.15.70:0/3792405480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78880aace0 0x7f7888140380 unknown :-1 s=CLOSED pgs=5754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 -- 172.21.15.70:0/3792405480 >> 172.21.15.70:0/3792405480 conn(0x7f7888067ba0 msgr2=0x7f7888067fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 -- 172.21.15.70:0/3792405480 shutdown_connections 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.966+0000 7f788fc27640 1 -- 172.21.15.70:0/3792405480 wait complete. 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 10 monclient: build_initial_monmap 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 10 monclient: monmap: 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 librados: starting msgr at 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 librados: starting objecter 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 Processor -- start 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 -- start start 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 librados: setting wanted keys 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 1 librados: calling monclient init 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.967+0000 7f788fc27640 10 monclient: init 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:21:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78880aace0 0x7f78880ad330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-b con 0x7f78880aace0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 0x7f78880ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-c con 0x7f78880b5e80 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 0x7f7888142130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): picked mon.noname-a con 0x7f78881408c0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7888146df0 con 0x7f78881408c0 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7888140d50 con 0x7f78880aace0 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f78880b3350 con 0x7f78880b5e80 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient(hunting): authenticate will time out at 2641.629227s 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 0x7f78880ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 0x7f7888142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 0x7f7888142130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33436/0 (socket says 172.21.15.70:33436) 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 -- 172.21.15.70:0/687994461 learned_addr learned my addr 172.21.15.70:0/687994461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 0x7f78880ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54904/0 (socket says 172.21.15.70:54904) 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): get_auth_request con 0x7f78881408c0 auth_method 0 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): get_auth_request con 0x7f78880b5e80 auth_method 0 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d9c3640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78880aace0 0x7f78880ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient(hunting): handle_auth_done global_id 26808 payload 824 2024-09-16T04:21:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 -- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 msgr2=0x7f78880ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 0x7f78880ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 -- 172.21.15.70:0/687994461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78880aace0 msgr2=0x7f78880ad330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78880aace0 0x7f78880ad330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient: _finish_hunting 0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 monclient: found mon.noname-a 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 -- 172.21.15.70:0/687994461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78880af160 con 0x7f78881408c0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d9c3640 10 monclient: get_auth_request con 0x7f78880aace0 auth_method 0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d9c3640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f78880aace0 0x7f78880ad330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788d1c2640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f78880b5e80 0x7f78880ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788e1c4640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 0x7f7888142130 secure :-1 s=READY pgs=5755 cs=0 l=1 rev1=1 crypto rx=0x7f788000efc0 tx=0x7f788000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 client.?.objecter ms_handle_connect 0x7f78881408c0 2024-09-16T04:21:22.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880018030 con 0x7f78881408c0 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: dump: 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.987 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788000cdf0 con 0x7f78881408c0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 5 monclient: authenticate success, global_id 26808 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _renew_subs 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78880af440 con 0x7f78881408c0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880003bb0 con 0x7f78881408c0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787e7fc640 10 monclient: dump: 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f787ffff640 4 set_mon_vals no callback set 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 client.26808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _renew_subs 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78880ad8f0 con 0x7f78881408c0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.968+0000 7f788fc27640 1 librados: init done 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 10 client.26808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 15 client.26808.objecter _session_command_op_assign -1 1 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 20 client.26808.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 10 client.26808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.969+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78880b54f0 con 0x7f78881408c0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f788001b070 con 0x7f78881408c0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.70:6804/198622594 conn(0x7f786c06dc00 0x7f786c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f788d9c3640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.70:6804/198622594 conn(0x7f786c06dc00 0x7f786c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f788008bb10 con 0x7f78881408c0 2024-09-16T04:21:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 10 client.26808.objecter ms_dispatch 0x7f7888000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 3 client.26808.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 3 client.26808.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f788d9c3640 10 monclient: get_auth_request con 0x7f786c06dc00 auth_method 0 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.970+0000 7f787e7fc640 10 client.26808.objecter checking command 1 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.163:6808/1625318059 conn(0x7f786c075520 0x7f786c0779c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter _get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 15 client.26808.objecter _session_command_op_remove -1 1 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter _get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 15 client.26808.objecter _session_command_op_assign 5 1 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 4 2024-09-16T04:21:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 10 client.26808.objecter _send_command 1 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 --> v2:172.21.15.163:6808/1625318059 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f788000be50 con 0x7f786c075520 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 20 client.26808.objecter dump_active .. 0 homeless 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7f78800902c0 con 0x7f78881408c0 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 10 client.26808.objecter ms_handle_connect 0x7f786c06dc00 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f788d9c3640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.70:6804/198622594 conn(0x7f786c06dc00 0x7f786c0700c0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f7884000f60 tx=0x7f7884009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f788d1c2640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.163:6808/1625318059 conn(0x7f786c075520 0x7f786c0779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f788d1c2640 10 monclient: get_auth_request con 0x7f786c075520 auth_method 0 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f787e7fc640 10 client.26808.objecter ms_handle_connect 0x7f786c075520 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.971+0000 7f788d1c2640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.163:6808/1625318059 conn(0x7f786c075520 0x7f786c0779c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.972+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== osd.5 v2:172.21.15.163:6808/1625318059 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f788000be50 con 0x7f786c075520 2024-09-16T04:21:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.972+0000 7f787e7fc640 10 client.26808.objecter ms_dispatch 0x7f7888000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.972+0000 7f787e7fc640 10 client.26808.objecter _finish_command 1 = system:0 2024-09-16T04:21:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.972+0000 7f787e7fc640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 4 2024-09-16T04:21:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.972+0000 7f787e7fc640 15 client.26808.objecter _session_command_op_remove 5 1 2024-09-16T04:21:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 10 client.26808.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 15 client.26808.objecter _session_command_op_assign -1 2 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 20 client.26808.objecter _get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 20 client.26808.objecter _get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 15 client.26808.objecter _session_command_op_remove -1 2 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 20 client.26808.objecter get_session s=0x7f786c074e70 osd=5 3 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 15 client.26808.objecter _session_command_op_assign 5 2 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 4 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 10 client.26808.objecter _send_command 2 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.984+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 --> v2:172.21.15.163:6808/1625318059 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f78880af7e0 con 0x7f786c075520 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f787e7fc640 1 -- 172.21.15.70:0/687994461 <== osd.5 v2:172.21.15.163:6808/1625318059 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f78880af7e0 con 0x7f786c075520 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f787e7fc640 10 client.26808.objecter ms_dispatch 0x7f7888000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f787e7fc640 10 client.26808.objecter _finish_command 2 = system:0 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f787e7fc640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 4 2024-09-16T04:21:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f787e7fc640 15 client.26808.objecter _session_command_op_remove 5 2 2024-09-16T04:21:22.995 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:22.995 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:22.995 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 10 librados: watch_flush enter 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 10 librados: watch_flush exit 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 10 client.26808.objecter close_session for osd.5 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 >> v2:172.21.15.163:6808/1625318059 conn(0x7f786c075520 msgr2=0x7f786c0779c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.163:6808/1625318059 conn(0x7f786c075520 0x7f786c0779c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 20 client.26808.objecter put_session s=0x7f786c074e70 osd=5 1 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 20 client.26808.objecter shutdown clearing up homeless session... 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 10 client.26808.objecter successfully canceled tick 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 >> v2:172.21.15.70:6804/198622594 conn(0x7f786c06dc00 msgr2=0x7f786c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 --2- 172.21.15.70:0/687994461 >> v2:172.21.15.70:6804/198622594 conn(0x7f786c06dc00 0x7f786c0700c0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f7884000f60 tx=0x7f7884009040 comp rx=0 tx=0).stop 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 10 monclient: shutdown 2024-09-16T04:21:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 msgr2=0x7f7888142130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 --2- 172.21.15.70:0/687994461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78881408c0 0x7f7888142130 secure :-1 s=READY pgs=5755 cs=0 l=1 rev1=1 crypto rx=0x7f788000efc0 tx=0x7f788000c490 comp rx=0 tx=0).stop 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788e1c4640 1 -- 172.21.15.70:0/687994461 reap_dead start 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 shutdown_connections 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 >> 172.21.15.70:0/687994461 conn(0x7f7888067ba0 msgr2=0x7f788813e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 shutdown_connections 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 -- 172.21.15.70:0/687994461 wait complete. 2024-09-16T04:21:22.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:22.985+0000 7f788fc27640 1 librados: shutdown 2024-09-16T04:21:22.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:21:23.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.095+0000 7ff9ddcf1640 10 monclient: get_monmap_and_config 2024-09-16T04:21:23.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.095+0000 7ff9ddcf1640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.095+0000 7ff9ddcf1640 10 monclient: monmap: 2024-09-16T04:21:23.096 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.096 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:23.095709+0000 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:23.095709+0000 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:23.097 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 Processor -- start 2024-09-16T04:21:23.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 -- start start 2024-09-16T04:21:23.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient: init 2024-09-16T04:21:23.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9d806c100 0x7ff9d80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d806c100 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9d81468a0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d81473e0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d80aebb0 con 0x7ff9d81468a0 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d806c4e0 con 0x7ff9d806c100 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d806c940 con 0x7ff9d81473e0 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9ddcf1640 10 monclient(hunting): authenticate will time out at 2641.757856s 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.096+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54916/0 (socket says 172.21.15.70:54916) 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 1 -- 172.21.15.70:0/3976850471 learned_addr learned my addr 172.21.15.70:0/3976850471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/3976850471 >> v2:172.21.15.70:3300/0 conn(0x7ff9d806c100 0x7ff9d80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9d81473e0 auth_method 0 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9cffff640 1 --1- 172.21.15.70:0/3976850471 >> v1:172.21.15.70:6789/0 conn(0x7ff9d81468a0 0x7ff9d8146cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42530/0 (socket says 172.21.15.70:42530) 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request con 0x7ff9d806c100 auth_method 0 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_done global_id 26811 payload 274 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 -- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 msgr2=0x7ff9d80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 -- 172.21.15.70:0/3976850471 >> v1:172.21.15.70:6789/0 conn(0x7ff9d81468a0 legacy=0x7ff9d8146cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 monclient: found mon.noname-a 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d77fe640 1 -- 172.21.15.70:0/3976850471 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d80aee20 con 0x7ff9d806c100 2024-09-16T04:21:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d6ffd640 1 -- 172.21.15.70:0/3976850471 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530909256 0 0) 0x7ff9d80aebb0 con 0x7ff9d81468a0 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d7fff640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.097+0000 7ff9d6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d80b1500 secure :-1 s=READY pgs=5757 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097d0 tx=0x7ff9c802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67482d8bca2d3f02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 1 -- 172.21.15.70:0/3976850471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8037070 con 0x7ff9d806c100 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: dump: 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 5 monclient: authenticate success, global_id 26811 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 1 -- 172.21.15.70:0/3976850471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c8032440 con 0x7ff9d806c100 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 4 set_mon_vals no callback set 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 1 -- 172.21.15.70:0/3976850471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8032810 con 0x7ff9d806c100 2024-09-16T04:21:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9d6ffd640 10 monclient: dump: 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 10 monclient: shutdown 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 msgr2=0x7ff9d80b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d80b1500 secure :-1 s=READY pgs=5757 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097d0 tx=0x7ff9c802a1c0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/3976850471 shutdown_connections 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/3976850471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d80b1500 unknown :-1 s=CLOSED pgs=5757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/3976850471 >> 172.21.15.70:0/3976850471 conn(0x7ff9d8068490 msgr2=0x7ff9d80b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/3976850471 shutdown_connections 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.098+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/3976850471 wait complete. 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 10 monclient: monmap: 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 librados: starting msgr at 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 librados: starting objecter 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 Processor -- start 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 -- start start 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 librados: setting wanted keys 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 1 librados: calling monclient init 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.099+0000 7ff9ddcf1640 10 monclient: init 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d806c100 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 0x7ff9d813ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d81468a0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d813cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d81473e0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80b1a40 con 0x7ff9d806c100 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80b11c0 con 0x7ff9d81468a0 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d806d150 con 0x7ff9d81473e0 2024-09-16T04:21:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9ddcf1640 10 monclient(hunting): authenticate will time out at 2641.761371s 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d813cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33446/0 (socket says 172.21.15.70:33446) 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 1 -- 172.21.15.70:0/417918966 learned_addr learned my addr 172.21.15.70:0/417918966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request con 0x7ff9d806c100 auth_method 0 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request con 0x7ff9d81473e0 auth_method 0 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient(hunting): handle_auth_done global_id 21014 payload 824 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9cffff640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 0x7ff9d813ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 -- 172.21.15.70:0/417918966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 msgr2=0x7ff9d813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 0x7ff9d813ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 -- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 msgr2=0x7ff9d811e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 monclient: found mon.noname-c 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 -- 172.21.15.70:0/417918966 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d813d490 con 0x7ff9d81473e0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9cffff640 10 monclient: get_auth_request con 0x7ff9d81468a0 auth_method 0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9cffff640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 0x7ff9d813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d7fff640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d813cf50 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009db0 tx=0x7ff9d000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d4ff9640 10 client.?.objecter ms_handle_connect 0x7ff9d81473e0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.100+0000 7ff9d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000cb20 con 0x7ff9d81473e0 2024-09-16T04:21:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: dump: 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d000cce0 con 0x7ff9d81473e0 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 5 monclient: authenticate success, global_id 21014 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _renew_subs 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d811fab0 con 0x7ff9d81473e0 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0003bd0 con 0x7ff9d81473e0 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d4ff9640 10 monclient: dump: 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 client.21014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _renew_subs 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9d67fc640 4 set_mon_vals no callback set 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d8143600 con 0x7ff9d81473e0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 1 librados: init done 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.101+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d80b1500 con 0x7ff9d81473e0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.102+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7ff9d001b030 con 0x7ff9d81473e0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.102+0000 7ff9d4ff9640 1 --2- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 0x7ff9b80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.102+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 0x7ff9b80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.102+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7ff9d008b480 con 0x7ff9d81473e0 2024-09-16T04:21:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d4ff9640 10 client.21014.objecter ms_dispatch 0x7ff9d8000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d4ff9640 3 client.21014.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d4ff9640 3 client.21014.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d77fe640 10 monclient: get_auth_request con 0x7ff9b806dc00 auth_method 0 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d4ff9640 20 client.21014.objecter dump_active .. 0 homeless 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d4ff9640 10 client.21014.objecter ms_handle_connect 0x7ff9b806dc00 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.103+0000 7ff9d77fe640 1 --2- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 0x7ff9b80700c0 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000c00 tx=0x7ff9c802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.107+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff9d0022020 con 0x7ff9d81473e0 2024-09-16T04:21:23.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.107+0000 7ff9d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.107+0000 7ff9d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:23.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.276+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7ff9d806de20 con 0x7ff9b806dc00 2024-09-16T04:21:23.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.276+0000 7ff9d4ff9640 1 -- 172.21.15.70:0/417918966 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9d806de20 con 0x7ff9b806dc00 2024-09-16T04:21:23.278 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:21:23.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.278+0000 7ff9ddcf1640 10 librados: watch_flush enter 2024-09-16T04:21:23.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.278+0000 7ff9ddcf1640 10 librados: watch_flush exit 2024-09-16T04:21:23.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.278+0000 7ff9ddcf1640 20 client.21014.objecter shutdown clearing up homeless session... 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.278+0000 7ff9ddcf1640 10 client.21014.objecter successfully canceled tick 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 msgr2=0x7ff9b80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 0x7ff9b80700c0 secure :-1 s=READY pgs=4039 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8000c00 tx=0x7ff9c802e000 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 10 monclient: shutdown 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 msgr2=0x7ff9d813cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d813cf50 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009db0 tx=0x7ff9d000c490 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 shutdown_connections 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> v2:172.21.15.70:6804/198622594 conn(0x7ff9b806dc00 0x7ff9b80700c0 unknown :-1 s=CLOSED pgs=4039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff9d81473e0 0x7ff9d813cf50 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff9d81468a0 0x7ff9d813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 --2- 172.21.15.70:0/417918966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9d806c100 0x7ff9d811e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 >> 172.21.15.70:0/417918966 conn(0x7ff9d8068490 msgr2=0x7ff9d80b3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 shutdown_connections 2024-09-16T04:21:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 -- 172.21.15.70:0/417918966 wait complete. 2024-09-16T04:21:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.279+0000 7ff9ddcf1640 1 librados: shutdown 2024-09-16T04:21:23.291 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T04:21:23.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.391+0000 7febbbf42640 10 monclient: get_monmap_and_config 2024-09-16T04:21:23.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.391+0000 7febbbf42640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.391+0000 7febbbf42640 10 monclient: monmap: 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:23.391809+0000 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:23.391809+0000 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:23.393 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:23.394 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.391+0000 7febbbf42640 1 Processor -- start 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 -- start start 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient: init 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febb40b5690 0x7febb41435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-a con 0x7febb40b5690 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 0x7febb4143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-c con 0x7febb40aac50 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:23.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febb40aa4e0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:23.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febb40ac240 con 0x7febb40aa4e0 2024-09-16T04:21:23.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7febb40b5a70 con 0x7febb40b5690 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7febb4142f50 con 0x7febb40aac50 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febbbf42640 10 monclient(hunting): authenticate will time out at 2642.053761s 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febba4df640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7febb40aa4e0 0x7febb41468b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42544/0 (socket says 172.21.15.70:42544) 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febb9cde640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febb40b5690 0x7febb41435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febba4df640 1 -- 172.21.15.70:0/3041801505 learned_addr learned my addr 172.21.15.70:0/3041801505 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febb9cde640 10 monclient(hunting): get_auth_request con 0x7febb40b5690 auth_method 0 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febb9cde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febb9cde640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.392+0000 7febb9cde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb94dd640 1 --2- 172.21.15.70:0/3041801505 >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 0x7febb4143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient(hunting): handle_auth_done global_id 26820 payload 274 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 -- 172.21.15.70:0/3041801505 >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 msgr2=0x7febb4143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 --2- 172.21.15.70:0/3041801505 >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 0x7febb4143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 -- 172.21.15.70:0/3041801505 >> v1:172.21.15.70:6789/0 conn(0x7febb40aa4e0 legacy=0x7febb41468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 monclient: found mon.noname-a 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 -- 172.21.15.70:0/3041801505 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb4146fc0 con 0x7febb40b5690 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb94dd640 10 monclient: get_auth_request con 0x7febb40aac50 auth_method 0 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb94dd640 1 --2- 172.21.15.70:0/3041801505 >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 0x7febb4143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb9cde640 1 --2- 172.21.15.70:0/3041801505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40b5690 0x7febb41435d0 secure :-1 s=READY pgs=5759 cs=0 l=1 rev1=1 crypto rx=0x7febb40b50f0 tx=0x7febac02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de390da06e7a8654 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb8cdc640 1 -- 172.21.15.70:0/3041801505 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac037070 con 0x7febb40b5690 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb8cdc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.393+0000 7febb8cdc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: dump: 2024-09-16T04:21:23.401 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.402 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 5 monclient: authenticate success, global_id 26820 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 1 -- 172.21.15.70:0/3041801505 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febac032440 con 0x7febb40b5690 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 4 set_mon_vals no callback set 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 1 -- 172.21.15.70:0/3041801505 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac02ae60 con 0x7febb40b5690 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febb8cdc640 10 monclient: dump: 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 10 monclient: shutdown 2024-09-16T04:21:23.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 -- 172.21.15.70:0/3041801505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40b5690 msgr2=0x7febb41435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 --2- 172.21.15.70:0/3041801505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40b5690 0x7febb41435d0 secure :-1 s=READY pgs=5759 cs=0 l=1 rev1=1 crypto rx=0x7febb40b50f0 tx=0x7febac02a110 comp rx=0 tx=0).stop 2024-09-16T04:21:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 -- 172.21.15.70:0/3041801505 shutdown_connections 2024-09-16T04:21:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 --2- 172.21.15.70:0/3041801505 >> v2:172.21.15.163:3300/0 conn(0x7febb40aac50 0x7febb4143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 --2- 172.21.15.70:0/3041801505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40b5690 0x7febb41435d0 unknown :-1 s=CLOSED pgs=5759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 -- 172.21.15.70:0/3041801505 >> 172.21.15.70:0/3041801505 conn(0x7febb40679c0 msgr2=0x7febb4067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 -- 172.21.15.70:0/3041801505 shutdown_connections 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.394+0000 7febbbf42640 1 -- 172.21.15.70:0/3041801505 wait complete. 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 10 monclient: monmap: 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:23.415 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 librados: starting msgr at 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 librados: starting objecter 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 Processor -- start 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 -- start start 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 librados: setting wanted keys 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 1 librados: calling monclient init 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.395+0000 7febbbf42640 10 monclient: init 2024-09-16T04:21:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7febb40aa4e0 0x7febb41d8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-c con 0x7febb40aa4e0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40aac50 0x7febb41d8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-a con 0x7febb40aac50 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febb40b5690 0x7febb40ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): picked mon.noname-b con 0x7febb40b5690 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febb414d9e0 con 0x7febb40aac50 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7febb414d860 con 0x7febb40b5690 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7febb414db60 con 0x7febb40aa4e0 2024-09-16T04:21:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient(hunting): authenticate will time out at 2642.057200s 2024-09-16T04:21:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7febb40aa4e0 0x7febb41d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40aac50 0x7febb41d8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7febb40aa4e0 0x7febb41d8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54930/0 (socket says 172.21.15.70:54930) 2024-09-16T04:21:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 1 -- 172.21.15.70:0/4090488514 learned_addr learned my addr 172.21.15.70:0/4090488514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): get_auth_request con 0x7febb40aac50 auth_method 0 2024-09-16T04:21:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): get_auth_request con 0x7febb40aa4e0 auth_method 0 2024-09-16T04:21:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb9cde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febba4df640 1 --2- 172.21.15.70:0/4090488514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febb40b5690 0x7febb40ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febba4df640 10 monclient(hunting): get_auth_request con 0x7febb40b5690 auth_method 0 2024-09-16T04:21:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febba4df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febba4df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febba4df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient(hunting): handle_auth_done global_id 26826 payload 824 2024-09-16T04:21:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 -- 172.21.15.70:0/4090488514 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7febb40aa4e0 msgr2=0x7febb41d8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 --2- 172.21.15.70:0/4090488514 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7febb40aa4e0 0x7febb41d8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 -- 172.21.15.70:0/4090488514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febb40b5690 msgr2=0x7febb40ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 --2- 172.21.15.70:0/4090488514 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febb40b5690 0x7febb40ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 monclient: found mon.noname-a 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 -- 172.21.15.70:0/4090488514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb40aea80 con 0x7febb40aac50 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb94dd640 1 --2- 172.21.15.70:0/4090488514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40aac50 0x7febb41d8d60 secure :-1 s=READY pgs=5760 cs=0 l=1 rev1=1 crypto rx=0x7febb4144050 tx=0x7feba800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 client.?.objecter ms_handle_connect 0x7febb40aac50 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8015070 con 0x7febb40aac50 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: dump: 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba8007820 con 0x7febb40aac50 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 5 monclient: authenticate success, global_id 26826 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _renew_subs 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb40aed60 con 0x7febb40aac50 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8007bd0 con 0x7febb40aac50 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb27fc640 10 monclient: dump: 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.431 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 client.26826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _renew_subs 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb414a400 con 0x7febb40aac50 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febbbf42640 1 librados: init done 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.396+0000 7febb3fff640 4 set_mon_vals no callback set 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 10 client.26826.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 15 client.26826.objecter _session_command_op_assign -1 1 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 20 client.26826.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:23.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 10 client.26826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.397+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7febb4146890 con 0x7febb40aac50 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7feba8018070 con 0x7febb40aac50 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.70:6804/198622594 conn(0x7feba006dc00 0x7feba00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb9cde640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.70:6804/198622594 conn(0x7feba006dc00 0x7feba00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7feba808b5d0 con 0x7febb40aac50 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 10 client.26826.objecter ms_dispatch 0x7febb4000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 3 client.26826.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb27fc640 3 client.26826.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.398+0000 7febb9cde640 10 monclient: get_auth_request con 0x7feba006dc00 auth_method 0 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 10 client.26826.objecter checking command 1 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.163:6812/2999981402 conn(0x7feba0075520 0x7feba00779c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter _get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 15 client.26826.objecter _session_command_op_remove -1 1 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter _get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 15 client.26826.objecter _session_command_op_assign 6 1 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 4 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 10 client.26826.objecter _send_command 1 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 --> v2:172.21.15.163:6812/2999981402 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feba0078050 con 0x7feba0075520 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 20 client.26826.objecter dump_active .. 0 homeless 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7feba808ba60 con 0x7febb40aac50 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:23.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 10 client.26826.objecter ms_handle_connect 0x7feba006dc00 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febba4df640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.163:6812/2999981402 conn(0x7feba0075520 0x7feba00779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb9cde640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.70:6804/198622594 conn(0x7feba006dc00 0x7feba00700c0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7febac009560 tx=0x7febac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febba4df640 10 monclient: get_auth_request con 0x7feba0075520 auth_method 0 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febb27fc640 10 client.26826.objecter ms_handle_connect 0x7feba0075520 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.399+0000 7febba4df640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.163:6812/2999981402 conn(0x7feba0075520 0x7feba00779c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.400+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== osd.6 v2:172.21.15.163:6812/2999981402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feba0078050 con 0x7feba0075520 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.400+0000 7febb27fc640 10 client.26826.objecter ms_dispatch 0x7febb4000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.400+0000 7febb27fc640 10 client.26826.objecter _finish_command 1 = system:0 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.400+0000 7febb27fc640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 4 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.400+0000 7febb27fc640 15 client.26826.objecter _session_command_op_remove 6 1 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 10 client.26826.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:23.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 15 client.26826.objecter _session_command_op_assign -1 2 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 20 client.26826.objecter _get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 20 client.26826.objecter _get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 15 client.26826.objecter _session_command_op_remove -1 2 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 20 client.26826.objecter get_session s=0x7feba0074e70 osd=6 3 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 15 client.26826.objecter _session_command_op_assign 6 2 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 4 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 10 client.26826.objecter _send_command 2 2024-09-16T04:21:23.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.412+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 --> v2:172.21.15.163:6812/2999981402 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7febb40adab0 con 0x7feba0075520 2024-09-16T04:21:23.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febb27fc640 1 -- 172.21.15.70:0/4090488514 <== osd.6 v2:172.21.15.163:6812/2999981402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7febb40adab0 con 0x7feba0075520 2024-09-16T04:21:23.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febb27fc640 10 client.26826.objecter ms_dispatch 0x7febb4000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:23.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febb27fc640 10 client.26826.objecter _finish_command 2 = system:0 2024-09-16T04:21:23.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febb27fc640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 4 2024-09-16T04:21:23.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febb27fc640 15 client.26826.objecter _session_command_op_remove 6 2 2024-09-16T04:21:23.440 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:23.440 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:23.440 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 10 librados: watch_flush enter 2024-09-16T04:21:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 10 librados: watch_flush exit 2024-09-16T04:21:23.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 10 client.26826.objecter close_session for osd.6 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 >> v2:172.21.15.163:6812/2999981402 conn(0x7feba0075520 msgr2=0x7feba00779c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.163:6812/2999981402 conn(0x7feba0075520 0x7feba00779c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 20 client.26826.objecter put_session s=0x7feba0074e70 osd=6 1 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 20 client.26826.objecter shutdown clearing up homeless session... 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 10 client.26826.objecter successfully canceled tick 2024-09-16T04:21:23.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 >> v2:172.21.15.70:6804/198622594 conn(0x7feba006dc00 msgr2=0x7feba00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 --2- 172.21.15.70:0/4090488514 >> v2:172.21.15.70:6804/198622594 conn(0x7feba006dc00 0x7feba00700c0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7febac009560 tx=0x7febac02e000 comp rx=0 tx=0).stop 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 10 monclient: shutdown 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40aac50 msgr2=0x7febb41d8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 --2- 172.21.15.70:0/4090488514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febb40aac50 0x7febb41d8d60 secure :-1 s=READY pgs=5760 cs=0 l=1 rev1=1 crypto rx=0x7febb4144050 tx=0x7feba800cc50 comp rx=0 tx=0).stop 2024-09-16T04:21:23.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febba4df640 1 -- 172.21.15.70:0/4090488514 reap_dead start 2024-09-16T04:21:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 shutdown_connections 2024-09-16T04:21:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 >> 172.21.15.70:0/4090488514 conn(0x7febb40679c0 msgr2=0x7febb40b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 shutdown_connections 2024-09-16T04:21:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.413+0000 7febbbf42640 1 -- 172.21.15.70:0/4090488514 wait complete. 2024-09-16T04:21:23.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.414+0000 7febbbf42640 1 librados: shutdown 2024-09-16T04:21:23.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T04:21:23.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 10 monclient: get_monmap_and_config 2024-09-16T04:21:23.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 10 monclient: monmap: 2024-09-16T04:21:23.545 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:23.544802+0000 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:23.544802+0000 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:23.546 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 1 Processor -- start 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 1 -- start start 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.544+0000 7ff8e47e8640 10 monclient: init 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8dc0b5470 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff8dc13cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 0x7ff8dc143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff8dc143040 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc0ac040 con 0x7ff8dc0b5470 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc0b5f60 con 0x7ff8dc143040 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc0ac2b0 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e47e8640 10 monclient(hunting): authenticate will time out at 2642.206748s 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54940/0 (socket says 172.21.15.70:54940) 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 1 -- 172.21.15.70:0/1479429635 learned_addr learned my addr 172.21.15.70:0/1479429635 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): get_auth_request con 0x7ff8dc13cbd0 auth_method 0 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.545+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_done global_id 21020 payload 274 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 -- 172.21.15.70:0/1479429635 >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 msgr2=0x7ff8dc143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e2d85640 1 --2- 172.21.15.70:0/1479429635 >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 0x7ff8dc143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 --2- 172.21.15.70:0/1479429635 >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 0x7ff8dc143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 -- 172.21.15.70:0/1479429635 >> v1:172.21.15.163:6789/0 conn(0x7ff8dc0b5470 legacy=0x7ff8dc0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 monclient: found mon.noname-b 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 -- 172.21.15.70:0/1479429635 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc0aaef0 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e2d85640 10 monclient: get_auth_request con 0x7ff8dc143040 auth_method 0 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e2d85640 1 --2- 172.21.15.70:0/1479429635 >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 0x7ff8dc143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:23.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1d83640 1 --2- 172.21.15.70:0/1479429635 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009870 tx=0x7ff8cc02a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1031b63d9ea0788a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1582640 1 -- 172.21.15.70:0/1479429635 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc037070 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1582640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1582640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.546+0000 7ff8e1582640 10 monclient: dump: 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.557 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 1 -- 172.21.15.70:0/1479429635 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8cc032440 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 5 monclient: authenticate success, global_id 21020 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 1 -- 172.21.15.70:0/1479429635 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc032810 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 4 set_mon_vals no callback set 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e1582640 10 monclient: dump: 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 10 monclient: shutdown 2024-09-16T04:21:23.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 -- 172.21.15.70:0/1479429635 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 msgr2=0x7ff8dc13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/1479429635 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009870 tx=0x7ff8cc02a2a0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 -- 172.21.15.70:0/1479429635 shutdown_connections 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/1479429635 >> v2:172.21.15.163:3300/0 conn(0x7ff8dc143040 0x7ff8dc143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/1479429635 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc13cbd0 0x7ff8dc13d020 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 -- 172.21.15.70:0/1479429635 >> 172.21.15.70:0/1479429635 conn(0x7ff8dc0679c0 msgr2=0x7ff8dc067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 -- 172.21.15.70:0/1479429635 shutdown_connections 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.547+0000 7ff8e47e8640 1 -- 172.21.15.70:0/1479429635 wait complete. 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 10 monclient: monmap: 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.560 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 librados: starting msgr at 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 librados: starting objecter 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 Processor -- start 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 -- start start 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 librados: setting wanted keys 2024-09-16T04:21:23.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 1 librados: calling monclient init 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.548+0000 7ff8e47e8640 10 monclient: init 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff8dc0b5470 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8dc13cbd0 0x7ff8dc13fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff8dc13cbd0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff8dc143040 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc143b50 con 0x7ff8dc0b5470 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc1439d0 con 0x7ff8dc13cbd0 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc143cd0 con 0x7ff8dc143040 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e47e8640 10 monclient(hunting): authenticate will time out at 2642.210547s 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:54956/0 (socket says 172.21.15.70:54956) 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33468/0 (socket says 172.21.15.70:33468) 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 -- 172.21.15.70:0/994289629 learned_addr learned my addr 172.21.15.70:0/994289629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): get_auth_request con 0x7ff8dc143040 auth_method 0 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): get_auth_request con 0x7ff8dc0b5470 auth_method 0 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8dc13cbd0 0x7ff8dc13fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): get_auth_request con 0x7ff8dc13cbd0 auth_method 0 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e1d83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient(hunting): handle_auth_done global_id 21023 payload 824 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 -- 172.21.15.70:0/994289629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8dc13cbd0 msgr2=0x7ff8dc13fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8dc13cbd0 0x7ff8dc13fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 -- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 msgr2=0x7ff8dc14bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 monclient: found mon.noname-c 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2d85640 1 -- 172.21.15.70:0/994289629 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc14b3d0 con 0x7ff8dc143040 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.549+0000 7ff8e2584640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e2d85640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c820 tx=0x7ff8d800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8dc143040 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8015070 con 0x7ff8dc143040 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: dump: 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.566 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 5 monclient: authenticate success, global_id 21023 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8007a30 con 0x7ff8dc143040 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 10 monclient: _renew_subs 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc14b6b0 con 0x7ff8dc143040 2024-09-16T04:21:23.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8007e20 con 0x7ff8dc143040 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8d2ffd640 10 monclient: dump: 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.568 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 10 client.21023.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e0d81640 4 set_mon_vals no callback set 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 10 monclient: _renew_subs 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc149fd0 con 0x7ff8dc143040 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.550+0000 7ff8e47e8640 1 librados: init done 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.551+0000 7ff8e47e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.551+0000 7ff8e47e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.551+0000 7ff8e47e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.551+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc13d020 con 0x7ff8dc143040 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.551+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7ff8d8018070 con 0x7ff8dc143040 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 1 --2- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 0x7ff8c00700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7ff8d808b060 con 0x7ff8dc143040 2024-09-16T04:21:23.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 10 client.21023.objecter ms_dispatch 0x7ff8dc000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 3 client.21023.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 3 client.21023.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8e2584640 1 --2- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 0x7ff8c00700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8e2584640 10 monclient: get_auth_request con 0x7ff8c006dc30 auth_method 0 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.552+0000 7ff8d2ffd640 20 client.21023.objecter dump_active .. 0 homeless 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.553+0000 7ff8d2ffd640 10 client.21023.objecter ms_handle_connect 0x7ff8c006dc30 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.553+0000 7ff8e2584640 1 --2- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 0x7ff8c00700f0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400b820 tx=0x7ff8d400d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.556+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff8d80593d0 con 0x7ff8dc143040 2024-09-16T04:21:23.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.557+0000 7ff8d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.557+0000 7ff8d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:23.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.727+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7ff8dc14d6a0 con 0x7ff8c006dc30 2024-09-16T04:21:23.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.727+0000 7ff8d2ffd640 1 -- 172.21.15.70:0/994289629 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff8dc14d6a0 con 0x7ff8c006dc30 2024-09-16T04:21:23.728 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 10 librados: watch_flush enter 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 10 librados: watch_flush exit 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 20 client.21023.objecter shutdown clearing up homeless session... 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 10 client.21023.objecter successfully canceled tick 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 msgr2=0x7ff8c00700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 0x7ff8c00700f0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400b820 tx=0x7ff8d400d040 comp rx=0 tx=0).stop 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 10 monclient: shutdown 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 msgr2=0x7ff8dc149330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800c820 tx=0x7ff8d800cce0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 shutdown_connections 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> v2:172.21.15.70:6804/198622594 conn(0x7ff8c006dc30 0x7ff8c00700f0 unknown :-1 s=CLOSED pgs=4041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7ff8dc143040 0x7ff8dc149330 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8dc13cbd0 0x7ff8dc13fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 --2- 172.21.15.70:0/994289629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8dc0b5470 0x7ff8dc14bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 >> 172.21.15.70:0/994289629 conn(0x7ff8dc0679c0 msgr2=0x7ff8dc0b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 shutdown_connections 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 -- 172.21.15.70:0/994289629 wait complete. 2024-09-16T04:21:23.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.729+0000 7ff8e47e8640 1 librados: shutdown 2024-09-16T04:21:23.741 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T04:21:23.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:21:23.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.841+0000 7f0dd90dd640 10 monclient: get_monmap_and_config 2024-09-16T04:21:23.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.841+0000 7f0dd90dd640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.841+0000 7f0dd90dd640 10 monclient: monmap: 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:23.842063+0000 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:23.842063+0000 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:23.843 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 Processor -- start 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 -- start start 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient: init 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f0dd406cbf0 0x7f0dd40aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd406cbf0 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dd406cfd0 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0dd40ab000 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd40abd60 con 0x7f0dd40ab000 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dd413d3c0 con 0x7f0dd406cfd0 2024-09-16T04:21:23.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f0dd40b3500 con 0x7f0dd406cbf0 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.842+0000 7f0dd90dd640 10 monclient(hunting): authenticate will time out at 2642.503994s 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0dd40ab000 0x7f0dd4142f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:42556/0 (socket says 172.21.15.70:42556) 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 1 -- 172.21.15.70:0/1332159989 learned_addr learned my addr 172.21.15.70:0/1332159989 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141993004 0 0) 0x7f0dd40abd60 con 0x7f0dd40ab000 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): _negotiate 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): my global_id is 26832 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc0003520 con 0x7f0dd40ab000 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/1332159989 >> v2:172.21.15.163:3300/0 conn(0x7f0dd406cbf0 0x7f0dd40aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f0dd406cbf0 auth_method 0 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3604387123 0 0) 0x7f0dc0003520 con 0x7f0dd40ab000 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 >> v2:172.21.15.163:3300/0 conn(0x7f0dd406cbf0 msgr2=0x7f0dd40aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 --2- 172.21.15.70:0/1332159989 >> v2:172.21.15.163:3300/0 conn(0x7f0dd406cbf0 0x7f0dd40aa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 >> v1:172.21.15.163:6789/0 conn(0x7f0dd406cfd0 legacy=0x7f0dd40b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd40abfd0 con 0x7f0dd40ab000 2024-09-16T04:21:23.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd90dd640 5 monclient: authenticate success, global_id 26832 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd90dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f0dcc0019f0 con 0x7f0dd40ab000 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: dump: 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.858 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 >> v1:172.21.15.70:6789/0 conn(0x7f0dd40ab000 legacy=0x7f0dd4142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): picked mon.a con 0x7f0dc00039b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): picked mon.b con 0x7f0dc0004800 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dc00052e0 0x7f0dc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): picked mon.c con 0x7f0dc00052e0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd40b3500 con 0x7f0dc00039b0 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc000c1b0 con 0x7f0dc0004800 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc000c3b0 con 0x7f0dc00052e0 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd259c640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd90dd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f0dc00039b0 auth_method 0 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dc00052e0 0x7f0dc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): get_auth_request con 0x7f0dc00052e0 auth_method 0 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dcbfff640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dcbfff640 10 monclient(hunting): get_auth_request con 0x7f0dc0004800 auth_method 0 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dcbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.843+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:23.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_done global_id 26832 payload 293 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dc00052e0 msgr2=0x7f0dc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dc00052e0 0x7f0dc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/1332159989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dc0004800 msgr2=0x7f0dc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 monclient: found mon.a 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd359e640 1 -- 172.21.15.70:0/1332159989 reap_dead start 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/1332159989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0003520 con 0x7f0dc00039b0 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dcbfff640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dc0004800 0x7f0dc0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dc0003dd0 secure :-1 s=READY pgs=5762 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400ba00 tx=0x7f0dc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8a5f1e22ad81e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4030070 con 0x7f0dc00039b0 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: dump: 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.864 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc402ad80 con 0x7f0dc00039b0 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd90dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd90dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 1 -- 172.21.15.70:0/1332159989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400b600 con 0x7f0dc00039b0 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd90dd640 4 set_mon_vals no callback set 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.844+0000 7f0dd259c640 10 monclient: dump: 2024-09-16T04:21:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 10 monclient: shutdown 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 -- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 msgr2=0x7f0dc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dc0003dd0 secure :-1 s=READY pgs=5762 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400ba00 tx=0x7f0dc0005830 comp rx=0 tx=0).stop 2024-09-16T04:21:23.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 -- 172.21.15.70:0/1332159989 shutdown_connections 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 --2- 172.21.15.70:0/1332159989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dc0003dd0 unknown :-1 s=CLOSED pgs=5762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 -- 172.21.15.70:0/1332159989 >> 172.21.15.70:0/1332159989 conn(0x7f0dd40b8ff0 msgr2=0x7f0dd40b9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 -- 172.21.15.70:0/1332159989 shutdown_connections 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.845+0000 7f0dd90dd640 1 -- 172.21.15.70:0/1332159989 wait complete. 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 10 monclient: monmap: 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 librados: starting msgr at 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 librados: starting objecter 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 Processor -- start 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 -- start start 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 librados: setting wanted keys 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 1 librados: calling monclient init 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.846+0000 7f0dd90dd640 10 monclient: init 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dd40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc00039b0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dd4140420 0x7f0dd40b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-b con 0x7f0dd4140420 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dd40b2460 0x7f0dd41dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0dd40b2460 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd40aa660 con 0x7f0dc00039b0 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd40aa4e0 con 0x7f0dd4140420 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f0dd40aa7e0 con 0x7f0dd40b2460 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient(hunting): authenticate will time out at 2642.508234s 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dd40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dd40b2460 0x7f0dd41dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dd40aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33486/0 (socket says 172.21.15.70:33486) 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/268870139 learned_addr learned my addr 172.21.15.70:0/268870139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f0dc00039b0 auth_method 0 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): get_auth_request con 0x7f0dd40b2460 auth_method 0 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcbfff640 1 --2- 172.21.15.70:0/268870139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dd4140420 0x7f0dd40b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcbfff640 10 monclient(hunting): get_auth_request con 0x7f0dd4140420 auth_method 0 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient(hunting): handle_auth_done global_id 26835 payload 824 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/268870139 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dd40b2460 msgr2=0x7f0dd41dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/268870139 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f0dd40b2460 0x7f0dd41dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/268870139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dd4140420 msgr2=0x7f0dd40b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/268870139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0dd4140420 0x7f0dd40b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 monclient: found mon.noname-a 2024-09-16T04:21:23.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 -- 172.21.15.70:0/268870139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd40ae1c0 con 0x7f0dc00039b0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd2d9d640 1 --2- 172.21.15.70:0/268870139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dd40aca20 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f0dc402a870 tx=0x7f0dc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 client.?.objecter ms_handle_connect 0x7f0dc00039b0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4043020 con 0x7f0dc00039b0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: dump: 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc403f680 con 0x7f0dc00039b0 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 5 monclient: authenticate success, global_id 26835 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _renew_subs 2024-09-16T04:21:23.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd40ae430 con 0x7f0dc00039b0 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400b600 con 0x7f0dc00039b0 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dcb7fe640 10 monclient: dump: 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.874 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 client.26835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _renew_subs 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd90dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.847+0000 7f0dd1d9b640 4 set_mon_vals no callback set 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd40b28e0 con 0x7f0dc00039b0 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 1 librados: init done 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 10 client.26835.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 15 client.26835.objecter _session_command_op_assign -1 1 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 20 client.26835.objecter _get_session osd=-1 returning homeless 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 10 client.26835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.848+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0dd40b2e80 con 0x7f0dc00039b0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f0dc4046020 con 0x7f0dc00039b0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.70:6804/198622594 conn(0x7f0db406dc00 0x7f0db40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcbfff640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.70:6804/198622594 conn(0x7f0db406dc00 0x7f0db40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f0dc40abd70 con 0x7f0dc00039b0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 10 client.26835.objecter ms_dispatch 0x7f0dd4000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 3 client.26835.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcb7fe640 3 client.26835.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.849+0000 7f0dcbfff640 10 monclient: get_auth_request con 0x7f0db406dc00 auth_method 0 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 10 client.26835.objecter checking command 1 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.163:6800/3710554613 conn(0x7f0db4075520 0x7f0db40779c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter _get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 15 client.26835.objecter _session_command_op_remove -1 1 2024-09-16T04:21:23.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter _get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 15 client.26835.objecter _session_command_op_assign 7 1 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 4 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 10 client.26835.objecter _send_command 1 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 --> v2:172.21.15.163:6800/3710554613 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0dc402ad80 con 0x7f0db4075520 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 20 client.26835.objecter dump_active .. 0 homeless 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=754) ==== 24+0+0 (secure 0 0 0) 0x7f0dc40ae470 con 0x7f0dc00039b0 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 10 monclient: handle_get_version_reply finishing 1 version 754 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 10 client.26835.objecter ms_handle_connect 0x7f0db406dc00 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcbfff640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.70:6804/198622594 conn(0x7f0db406dc00 0x7f0db40700c0 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000c00 tx=0x7f0dbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dd359e640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.163:6800/3710554613 conn(0x7f0db4075520 0x7f0db40779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dd359e640 10 monclient: get_auth_request con 0x7f0db4075520 auth_method 0 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dcb7fe640 10 client.26835.objecter ms_handle_connect 0x7f0db4075520 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.850+0000 7f0dd359e640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.163:6800/3710554613 conn(0x7f0db4075520 0x7f0db40779c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.851+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== osd.7 v2:172.21.15.163:6800/3710554613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0dc402ad80 con 0x7f0db4075520 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.851+0000 7f0dcb7fe640 10 client.26835.objecter ms_dispatch 0x7f0dd4000b90 command_reply(tid 1: 0 ) 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.851+0000 7f0dcb7fe640 10 client.26835.objecter _finish_command 1 = system:0 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.851+0000 7f0dcb7fe640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 4 2024-09-16T04:21:23.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.851+0000 7f0dcb7fe640 15 client.26835.objecter _session_command_op_remove 7 1 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 10 client.26835.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 15 client.26835.objecter _session_command_op_assign -1 2 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 20 client.26835.objecter _get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 20 client.26835.objecter _get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 15 client.26835.objecter _session_command_op_remove -1 2 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 20 client.26835.objecter get_session s=0x7f0db4074e70 osd=7 3 2024-09-16T04:21:23.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 15 client.26835.objecter _session_command_op_assign 7 2 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 4 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 10 client.26835.objecter _send_command 2 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.863+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 --> v2:172.21.15.163:6800/3710554613 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0dd41408e0 con 0x7f0db4075520 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dcb7fe640 1 -- 172.21.15.70:0/268870139 <== osd.7 v2:172.21.15.163:6800/3710554613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0dd41408e0 con 0x7f0db4075520 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dcb7fe640 10 client.26835.objecter ms_dispatch 0x7f0dd4000b90 command_reply(tid 2: 0 ) 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dcb7fe640 10 client.26835.objecter _finish_command 2 = system:0 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dcb7fe640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 4 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dcb7fe640 15 client.26835.objecter _session_command_op_remove 7 2 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-16T04:21:23.880 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:21:23.881 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-16T04:21:23.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 10 librados: watch_flush enter 2024-09-16T04:21:23.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 10 librados: watch_flush exit 2024-09-16T04:21:23.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 10 client.26835.objecter close_session for osd.7 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 >> v2:172.21.15.163:6800/3710554613 conn(0x7f0db4075520 msgr2=0x7f0db40779c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.163:6800/3710554613 conn(0x7f0db4075520 0x7f0db40779c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 20 client.26835.objecter put_session s=0x7f0db4074e70 osd=7 1 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 20 client.26835.objecter shutdown clearing up homeless session... 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 10 client.26835.objecter successfully canceled tick 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 >> v2:172.21.15.70:6804/198622594 conn(0x7f0db406dc00 msgr2=0x7f0db40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 --2- 172.21.15.70:0/268870139 >> v2:172.21.15.70:6804/198622594 conn(0x7f0db406dc00 0x7f0db40700c0 secure :-1 s=READY pgs=4042 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000c00 tx=0x7f0dbc009040 comp rx=0 tx=0).stop 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 10 monclient: shutdown 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 msgr2=0x7f0dd40aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 --2- 172.21.15.70:0/268870139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc00039b0 0x7f0dd40aca20 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f0dc402a870 tx=0x7f0dc4004730 comp rx=0 tx=0).stop 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd359e640 1 -- 172.21.15.70:0/268870139 reap_dead start 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 shutdown_connections 2024-09-16T04:21:23.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 >> 172.21.15.70:0/268870139 conn(0x7f0dd40b8ff0 msgr2=0x7f0dd4142120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:23.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.864+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 shutdown_connections 2024-09-16T04:21:23.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.865+0000 7f0dd90dd640 1 -- 172.21.15.70:0/268870139 wait complete. 2024-09-16T04:21:23.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.865+0000 7f0dd90dd640 1 librados: shutdown 2024-09-16T04:21:23.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T04:21:23.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.982+0000 7f5f71349640 10 monclient: get_monmap_and_config 2024-09-16T04:21:23.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.982+0000 7f5f71349640 10 monclient: build_initial_monmap 2024-09-16T04:21:23.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.982+0000 7f5f71349640 10 monclient: monmap: 2024-09-16T04:21:23.983 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:23.982911+0000 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:23.982911+0000 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:23.984 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.982+0000 7f5f71349640 1 Processor -- start 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 -- start start 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient: init 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c0b37a0 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f6c0b40c0 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f5f6c1430f0 0x7f5f6c143580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c1430f0 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6c0abcf0 con 0x7f5f6c0b40c0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c0abf60 con 0x7f5f6c0b37a0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c0ac240 con 0x7f5f6c1430f0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f71349640 10 monclient(hunting): authenticate will time out at 2642.644689s 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33494/0 (socket says 172.21.15.70:33494) 2024-09-16T04:21:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 1 -- 172.21.15.70:0/12354166 learned_addr learned my addr 172.21.15.70:0/12354166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): get_auth_request con 0x7f5f6c0b37a0 auth_method 0 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6b7fe640 1 --2- 172.21.15.70:0/12354166 >> v2:172.21.15.163:3300/0 conn(0x7f5f6c1430f0 0x7f5f6c143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f6c1430f0 auth_method 0 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.983+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 10 monclient(hunting): handle_auth_done global_id 26838 payload 274 2024-09-16T04:21:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 -- 172.21.15.70:0/12354166 >> v2:172.21.15.163:3300/0 conn(0x7f5f6c1430f0 msgr2=0x7f5f6c143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 --2- 172.21.15.70:0/12354166 >> v2:172.21.15.163:3300/0 conn(0x7f5f6c1430f0 0x7f5f6c143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 -- 172.21.15.70:0/12354166 >> v1:172.21.15.163:6789/0 conn(0x7f5f6c0b40c0 legacy=0x7f5f6c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 monclient: found mon.noname-a 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 -- 172.21.15.70:0/12354166 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c0b4590 con 0x7f5f6c0b37a0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6affd640 1 --2- 172.21.15.70:0/12354166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0ab340 tx=0x7f5f5c02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a487c0df0963c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 1 -- 172.21.15.70:0/12354166 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c037070 con 0x7f5f6c0b37a0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: dump: 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:23.998 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 1 -- 172.21.15.70:0/12354166 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c032440 con 0x7f5f6c0b37a0 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 1 -- 172.21.15.70:0/12354166 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c032810 con 0x7f5f6c0b37a0 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f71349640 5 monclient: authenticate success, global_id 26838 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f71349640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f71349640 4 set_mon_vals no callback set 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.984+0000 7f5f6a7fc640 10 monclient: dump: 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 10 monclient: shutdown 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 -- 172.21.15.70:0/12354166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c0b37a0 msgr2=0x7f5f6c0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 --2- 172.21.15.70:0/12354166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0ab340 tx=0x7f5f5c02a150 comp rx=0 tx=0).stop 2024-09-16T04:21:24.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 -- 172.21.15.70:0/12354166 shutdown_connections 2024-09-16T04:21:24.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 --2- 172.21.15.70:0/12354166 >> v2:172.21.15.163:3300/0 conn(0x7f5f6c1430f0 0x7f5f6c143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 --2- 172.21.15.70:0/12354166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c0b3b80 unknown :-1 s=CLOSED pgs=5764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 -- 172.21.15.70:0/12354166 >> 172.21.15.70:0/12354166 conn(0x7f5f6c0679c0 msgr2=0x7f5f6c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:24.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 -- 172.21.15.70:0/12354166 shutdown_connections 2024-09-16T04:21:24.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.985+0000 7f5f71349640 1 -- 172.21.15.70:0/12354166 wait complete. 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 10 monclient: build_initial_monmap 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 10 monclient: monmap: 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 librados: starting msgr at 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 librados: starting objecter 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 Processor -- start 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 -- start start 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 librados: setting wanted keys 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 1 librados: calling monclient init 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.986+0000 7f5f71349640 10 monclient: init 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c13d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c0b37a0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5f6c0b40c0 0x7f5f6c13e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c0b40c0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c1430f0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c143c40 con 0x7f5f6c1430f0 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c143ac0 con 0x7f5f6c0b37a0 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c143dc0 con 0x7f5f6c0b40c0 2024-09-16T04:21:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f71349640 10 monclient(hunting): authenticate will time out at 2642.648288s 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33498/0 (socket says 172.21.15.70:33498) 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 -- 172.21.15.70:0/4218468059 learned_addr learned my addr 172.21.15.70:0/4218468059 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f6c1430f0 auth_method 0 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5f6c0b40c0 0x7f5f6c13e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c13d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): get_auth_request con 0x7f5f6c0b40c0 auth_method 0 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): get_auth_request con 0x7f5f6c0b37a0 auth_method 0 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient(hunting): handle_auth_done global_id 26841 payload 824 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 -- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5f6c0b40c0 msgr2=0x7f5f6c13e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5f6c0b40c0 0x7f5f6c13e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 -- 172.21.15.70:0/4218468059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f6c0b37a0 msgr2=0x7f5f6c13d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c13d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 -- 172.21.15.70:0/4218468059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c13f150 con 0x7f5f6c1430f0 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f6b7fe640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 secure :-1 s=READY pgs=5765 cs=0 l=1 rev1=1 crypto rx=0x7f5f64007e90 tx=0x7f5f6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 10 client.?.objecter ms_handle_connect 0x7f5f6c1430f0 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:24.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64014420 con 0x7f5f6c1430f0 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.987+0000 7f5f63fff640 10 monclient: dump: 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.009 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 5 monclient: authenticate success, global_id 26841 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _renew_subs 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f64019070 con 0x7f5f6c1430f0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c13f3c0 con 0x7f5f6c1430f0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64014800 con 0x7f5f6c1430f0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f63fff640 10 monclient: dump: 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.010 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f69ffb640 4 set_mon_vals no callback set 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 client.26841.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _renew_subs 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c13f980 con 0x7f5f6c1430f0 2024-09-16T04:21:24.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 1 librados: init done 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.988+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c0b3b80 con 0x7f5f6c1430f0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f5f640149a0 con 0x7f5f6c1430f0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 1 --2- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 0x7f5f4c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f6affd640 1 --2- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 0x7f5f4c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f5f6408b980 con 0x7f5f6c1430f0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 10 client.26841.objecter ms_dispatch 0x7f5f6c000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 3 client.26841.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f63fff640 3 client.26841.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.989+0000 7f5f6affd640 10 monclient: get_auth_request con 0x7f5f4c06dc00 auth_method 0 2024-09-16T04:21:24.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.990+0000 7f5f63fff640 20 client.26841.objecter dump_active .. 0 homeless 2024-09-16T04:21:24.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.990+0000 7f5f63fff640 10 client.26841.objecter ms_handle_connect 0x7f5f4c06dc00 2024-09-16T04:21:24.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.990+0000 7f5f6affd640 1 --2- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 0x7f5f4c0700c0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c034000 tx=0x7f5f5c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:24.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.994+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5f64059c60 con 0x7f5f6c1430f0 2024-09-16T04:21:24.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.994+0000 7f5f63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:23.994+0000 7f5f63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:24.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.160+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f5f6c1413e0 con 0x7f5f4c06dc00 2024-09-16T04:21:24.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.161+0000 7f5f63fff640 1 -- 172.21.15.70:0/4218468059 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5f6c1413e0 con 0x7f5f4c06dc00 2024-09-16T04:21:24.162 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T04:21:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 10 librados: watch_flush enter 2024-09-16T04:21:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 10 librados: watch_flush exit 2024-09-16T04:21:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 20 client.26841.objecter shutdown clearing up homeless session... 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 10 client.26841.objecter successfully canceled tick 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 msgr2=0x7f5f4c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 0x7f5f4c0700c0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c034000 tx=0x7f5f5c02e000 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 10 monclient: shutdown 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 msgr2=0x7f5f6c1449e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 secure :-1 s=READY pgs=5765 cs=0 l=1 rev1=1 crypto rx=0x7f5f64007e90 tx=0x7f5f6400e4b0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 shutdown_connections 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.162+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> v2:172.21.15.70:6804/198622594 conn(0x7f5f4c06dc00 0x7f5f4c0700c0 unknown :-1 s=CLOSED pgs=4043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f6c1430f0 0x7f5f6c1449e0 unknown :-1 s=CLOSED pgs=5765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f5f6c0b40c0 0x7f5f6c13e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 --2- 172.21.15.70:0/4218468059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f6c0b37a0 0x7f5f6c13d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 >> 172.21.15.70:0/4218468059 conn(0x7f5f6c0679c0 msgr2=0x7f5f6c0b94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:24.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 shutdown_connections 2024-09-16T04:21:24.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 -- 172.21.15.70:0/4218468059 wait complete. 2024-09-16T04:21:24.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.163+0000 7f5f71349640 1 librados: shutdown 2024-09-16T04:21:24.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:21:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.271+0000 7f875ed98640 10 monclient: get_monmap_and_config 2024-09-16T04:21:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.271+0000 7f875ed98640 10 monclient: build_initial_monmap 2024-09-16T04:21:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.271+0000 7f875ed98640 10 monclient: monmap: 2024-09-16T04:21:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:24.272 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:24.271759+0000 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:24.271759+0000 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:24.273 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 Processor -- start 2024-09-16T04:21:24.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 -- start start 2024-09-16T04:21:24.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient: init 2024-09-16T04:21:24.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f875806c1a0 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87580b1570 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f87580afd60 0x7f87580b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-c con 0x7f87580afd60 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87580b11c0 con 0x7f875806c1a0 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87580aef50 con 0x7f87580b1570 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f875806c580 con 0x7f87580afd60 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875ed98640 10 monclient(hunting): authenticate will time out at 2642.933743s 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875cb34640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f875806c1a0 0x7f875806d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:41244/0 (socket says 172.21.15.70:41244) 2024-09-16T04:21:24.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.272+0000 7f875cb34640 1 -- 172.21.15.70:0/3731756559 learned_addr learned my addr 172.21.15.70:0/3731756559 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1002598943 0 0) 0x7f87580b11c0 con 0x7f875806c1a0 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): my global_id is 26844 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8748003520 con 0x7f875806c1a0 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 1 --2- 172.21.15.70:0/3731756559 >> v2:172.21.15.163:3300/0 conn(0x7f87580afd60 0x7f87580b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): get_auth_request con 0x7f87580afd60 auth_method 0 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2116827370 0 0) 0x7f8748003520 con 0x7f875806c1a0 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 >> v2:172.21.15.163:3300/0 conn(0x7f87580afd60 msgr2=0x7f87580b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 --2- 172.21.15.70:0/3731756559 >> v2:172.21.15.163:3300/0 conn(0x7f87580afd60 0x7f87580b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 >> v1:172.21.15.163:6789/0 conn(0x7f87580b1570 legacy=0x7f87580af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.70:6789/0 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87580af230 con 0x7f875806c1a0 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _finish_auth 0 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:24.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875ed98640 5 monclient: authenticate success, global_id 26844 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3648168913 0 0) 0x7f874c001670 con 0x7f875806c1a0 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875ed98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: dump: 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.287 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] but i'm connected to v1:172.21.15.70:6789/0, reconnecting 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 >> v1:172.21.15.70:6789/0 conn(0x7f875806c1a0 legacy=0x7f875806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f87480039b0 0x7f8748003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): picked mon.c con 0x7f87480039b0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f8748004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): picked mon.a con 0x7f8748004800 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87480052e0 0x7f874800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): picked mon.b con 0x7f87480052e0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:24.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f875806c580 con 0x7f8748004800 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f874800c1b0 con 0x7f87480052e0 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f874800c3b0 con 0x7f87480039b0 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f87577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875ed98640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f87480039b0 0x7f8748003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): get_auth_request con 0x7f87480039b0 auth_method 0 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f8748004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87480052e0 0x7f874800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): get_auth_request con 0x7f8748004800 auth_method 0 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): get_auth_request con 0x7f87480052e0 auth_method 0 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient(hunting): handle_auth_done global_id 26844 payload 293 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 -- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f87480039b0 msgr2=0x7f8748003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f87480039b0 0x7f8748003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 -- 172.21.15.70:0/3731756559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87480052e0 msgr2=0x7f874800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87480052e0 0x7f874800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:24.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 monclient: found mon.a 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 1 -- 172.21.15.70:0/3731756559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8748003520 con 0x7f8748004800 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f875d335640 1 -- 172.21.15.70:0/3731756559 reap_dead start 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.273+0000 7f8757fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f8757fff640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f8748004c80 secure :-1 s=READY pgs=5767 cs=0 l=1 rev1=1 crypto rx=0x7f8744001720 tx=0x7f8748005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c657aaa431c2a92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744001370 con 0x7f8748004800 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: dump: 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.292 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f875ed98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87440052d0 con 0x7f8748004800 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 1 -- 172.21.15.70:0/3731756559 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874402ab90 con 0x7f8748004800 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f875ed98640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:24.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f875ed98640 4 set_mon_vals no callback set 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.274+0000 7f87577fe640 10 monclient: dump: 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.294 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 10 monclient: shutdown 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 -- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 msgr2=0x7f8748004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f8748004c80 secure :-1 s=READY pgs=5767 cs=0 l=1 rev1=1 crypto rx=0x7f8744001720 tx=0x7f8748005830 comp rx=0 tx=0).stop 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 -- 172.21.15.70:0/3731756559 shutdown_connections 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 --2- 172.21.15.70:0/3731756559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f8748004c80 unknown :-1 s=CLOSED pgs=5767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 -- 172.21.15.70:0/3731756559 >> 172.21.15.70:0/3731756559 conn(0x7f875806f030 msgr2=0x7f875806f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 -- 172.21.15.70:0/3731756559 shutdown_connections 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 1 -- 172.21.15.70:0/3731756559 wait complete. 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 10 monclient: build_initial_monmap 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.275+0000 7f875ed98640 10 monclient: monmap: 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:24.295 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 librados: starting msgr at 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 librados: starting objecter 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 Processor -- start 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 -- start start 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 librados: setting wanted keys 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 librados: calling monclient init 2024-09-16T04:21:24.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 10 monclient: init 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.276+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-a con 0x7f8748004800 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 0x7f87581430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-c con 0x7f8758143e30 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 0x7f87581d8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): picked mon.noname-b con 0x7f87581d84a0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f875806d250 con 0x7f8748004800 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f875806d0d0 con 0x7f87581d84a0 2024-09-16T04:21:24.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f875806d3d0 con 0x7f8758143e30 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient(hunting): authenticate will time out at 2642.938053s 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 0x7f87581430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53928/0 (socket says 172.21.15.70:53928) 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 -- 172.21.15.70:0/3827983440 learned_addr learned my addr 172.21.15.70:0/3827983440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): get_auth_request con 0x7f8748004800 auth_method 0 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): get_auth_request con 0x7f8758143e30 auth_method 0 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:24.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875d335640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 0x7f87581d8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient(hunting): handle_auth_done global_id 26847 payload 824 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 -- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 msgr2=0x7f87581430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 0x7f87581430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 -- 172.21.15.70:0/3827983440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 msgr2=0x7f87581d8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 0x7f87581d8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient: _finish_hunting 0 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 monclient: found mon.noname-a 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 -- 172.21.15.70:0/3827983440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8758143670 con 0x7f8748004800 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875d335640 10 monclient: get_auth_request con 0x7f87581d84a0 auth_method 0 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875d335640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 0x7f87581d8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8757fff640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 0x7f87581430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875cb34640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 crypto rx=0x7f874c004ec0 tx=0x7f874c007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 client.?.objecter ms_handle_connect 0x7f8748004800 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c00f7b0 con 0x7f8748004800 2024-09-16T04:21:24.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: dump: 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: _finish_auth 0 2024-09-16T04:21:24.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: _check_auth_tickets 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 5 monclient: authenticate success, global_id 26847 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c018070 con 0x7f8748004800 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c00fbb0 con 0x7f8748004800 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient: _renew_subs 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875813a7f0 con 0x7f8748004800 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f8756ffd640 4 set_mon_vals no callback set 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f87557fa640 10 monclient: dump: 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:24.302 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 client.26847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient: _renew_subs 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f875813c210 con 0x7f8748004800 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.277+0000 7f875ed98640 1 librados: init done 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.278+0000 7f875ed98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.278+0000 7f875ed98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.278+0000 7f875ed98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.70:3300/0 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.278+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87580af650 con 0x7f8748004800 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f874c00fd50 con 0x7f8748004800 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 1 --2- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 0x7f873c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f8757fff640 1 --2- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 0x7f873c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f874c08b110 con 0x7f8748004800 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 10 client.26847.objecter ms_dispatch 0x7f8758000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 3 client.26847.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 3 client.26847.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f8757fff640 10 monclient: get_auth_request con 0x7f873c06dc00 auth_method 0 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.279+0000 7f87557fa640 20 client.26847.objecter dump_active .. 0 homeless 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.280+0000 7f87557fa640 10 client.26847.objecter ms_handle_connect 0x7f873c06dc00 2024-09-16T04:21:24.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.280+0000 7f8757fff640 1 --2- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 0x7f873c0700c0 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7f8744032000 tx=0x7f874402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:24.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.284+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f874c059ad0 con 0x7f8748004800 2024-09-16T04:21:24.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.284+0000 7f87557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:24.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.284+0000 7f87557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:24.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.446+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f875806de20 con 0x7f873c06dc00 2024-09-16T04:21:24.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.449+0000 7f87557fa640 1 -- 172.21.15.70:0/3827983440 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116042 (secure 0 0 0) 0x7f875806de20 con 0x7f873c06dc00 2024-09-16T04:21:24.451 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:24.452 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 10 librados: watch_flush enter 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 10 librados: watch_flush exit 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 20 client.26847.objecter shutdown clearing up homeless session... 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 10 client.26847.objecter successfully canceled tick 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 msgr2=0x7f873c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 0x7f873c0700c0 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7f8744032000 tx=0x7f874402e000 comp rx=0 tx=0).stop 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 10 monclient: shutdown 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 msgr2=0x7f87580b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 crypto rx=0x7f874c004ec0 tx=0x7f874c007430 comp rx=0 tx=0).stop 2024-09-16T04:21:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 shutdown_connections 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> v2:172.21.15.70:6804/198622594 conn(0x7f873c06dc00 0x7f873c0700c0 unknown :-1 s=CLOSED pgs=4044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87581d84a0 0x7f87581d8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8758143e30 0x7f87581430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 --2- 172.21.15.70:0/3827983440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8748004800 0x7f87580b5f50 unknown :-1 s=CLOSED pgs=5768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 >> 172.21.15.70:0/3827983440 conn(0x7f875806f030 msgr2=0x7f875813f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 shutdown_connections 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 -- 172.21.15.70:0/3827983440 wait complete. 2024-09-16T04:21:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:24.451+0000 7f875ed98640 1 librados: shutdown 2024-09-16T04:21:24.464 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1696,"stamp":"2024-09-16T04:21:23.173634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":586092,"kb_used_data":32060,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754388628,"statfs":{"total":773094113280,"available":772493955072,"internally_reserved":0,"allocated":32829440,"data_stored":14997820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":563793458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.305478"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1440,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.082501+0000","last_change":"2024-09-16T03:50:58.527921+0000","last_active":"2024-09-16T04:21:18.082501+0000","last_peered":"2024-09-16T04:21:18.082501+0000","last_clean":"2024-09-16T04:21:18.082501+0000","last_became_active":"2024-09-16T03:50:58.527591+0000","last_became_peered":"2024-09-16T03:50:58.527591+0000","last_unstale":"2024-09-16T04:21:18.082501+0000","last_undegraded":"2024-09-16T04:21:18.082501+0000","last_fullsized":"2024-09-16T04:21:18.082501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:51.155864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1464,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.318848+0000","last_change":"2024-09-16T03:50:58.566215+0000","last_active":"2024-09-16T04:21:18.318848+0000","last_peered":"2024-09-16T04:21:18.318848+0000","last_clean":"2024-09-16T04:21:18.318848+0000","last_became_active":"2024-09-16T03:50:58.565169+0000","last_became_peered":"2024-09-16T03:50:58.565169+0000","last_unstale":"2024-09-16T04:21:18.318848+0000","last_undegraded":"2024-09-16T04:21:18.318848+0000","last_fullsized":"2024-09-16T04:21:18.318848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:31.209733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1464,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.318620+0000","last_change":"2024-09-16T03:50:58.521189+0000","last_active":"2024-09-16T04:21:18.318620+0000","last_peered":"2024-09-16T04:21:18.318620+0000","last_clean":"2024-09-16T04:21:18.318620+0000","last_became_active":"2024-09-16T03:50:58.520920+0000","last_became_peered":"2024-09-16T03:50:58.520920+0000","last_unstale":"2024-09-16T04:21:18.318620+0000","last_undegraded":"2024-09-16T04:21:18.318620+0000","last_fullsized":"2024-09-16T04:21:18.318620+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:15.187842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1440,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.083110+0000","last_change":"2024-09-16T03:50:58.529897+0000","last_active":"2024-09-16T04:21:18.083110+0000","last_peered":"2024-09-16T04:21:18.083110+0000","last_clean":"2024-09-16T04:21:18.083110+0000","last_became_active":"2024-09-16T03:50:58.529616+0000","last_became_peered":"2024-09-16T03:50:58.529616+0000","last_unstale":"2024-09-16T04:21:18.083110+0000","last_undegraded":"2024-09-16T04:21:18.083110+0000","last_fullsized":"2024-09-16T04:21:18.083110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:40.902750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1452,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316328+0000","last_change":"2024-09-16T03:50:58.547657+0000","last_active":"2024-09-16T04:21:18.316328+0000","last_peered":"2024-09-16T04:21:18.316328+0000","last_clean":"2024-09-16T04:21:18.316328+0000","last_became_active":"2024-09-16T03:50:58.547418+0000","last_became_peered":"2024-09-16T03:50:58.547418+0000","last_unstale":"2024-09-16T04:21:18.316328+0000","last_undegraded":"2024-09-16T04:21:18.316328+0000","last_fullsized":"2024-09-16T04:21:18.316328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:45.151909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.880284+0000","last_change":"2024-09-16T03:50:58.563918+0000","last_active":"2024-09-16T04:21:17.880284+0000","last_peered":"2024-09-16T04:21:17.880284+0000","last_clean":"2024-09-16T04:21:17.880284+0000","last_became_active":"2024-09-16T03:50:58.563681+0000","last_became_peered":"2024-09-16T03:50:58.563681+0000","last_unstale":"2024-09-16T04:21:17.880284+0000","last_undegraded":"2024-09-16T04:21:17.880284+0000","last_fullsized":"2024-09-16T04:21:17.880284+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:01.363054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1450,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.880367+0000","last_change":"2024-09-16T03:50:58.563162+0000","last_active":"2024-09-16T04:21:17.880367+0000","last_peered":"2024-09-16T04:21:17.880367+0000","last_clean":"2024-09-16T04:21:17.880367+0000","last_became_active":"2024-09-16T03:50:58.562917+0000","last_became_peered":"2024-09-16T03:50:58.562917+0000","last_unstale":"2024-09-16T04:21:17.880367+0000","last_undegraded":"2024-09-16T04:21:17.880367+0000","last_fullsized":"2024-09-16T04:21:17.880367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:20.514696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881328+0000","last_change":"2024-09-16T03:50:58.552412+0000","last_active":"2024-09-16T04:21:17.881328+0000","last_peered":"2024-09-16T04:21:17.881328+0000","last_clean":"2024-09-16T04:21:17.881328+0000","last_became_active":"2024-09-16T03:50:58.552137+0000","last_became_peered":"2024-09-16T03:50:58.552137+0000","last_unstale":"2024-09-16T04:21:17.881328+0000","last_undegraded":"2024-09-16T04:21:17.881328+0000","last_fullsized":"2024-09-16T04:21:17.881328+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:13.498957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1450,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.880398+0000","last_change":"2024-09-16T03:50:58.546028+0000","last_active":"2024-09-16T04:21:17.880398+0000","last_peered":"2024-09-16T04:21:17.880398+0000","last_clean":"2024-09-16T04:21:17.880398+0000","last_became_active":"2024-09-16T03:50:58.545776+0000","last_became_peered":"2024-09-16T03:50:58.545776+0000","last_unstale":"2024-09-16T04:21:17.880398+0000","last_undegraded":"2024-09-16T04:21:17.880398+0000","last_fullsized":"2024-09-16T04:21:17.880398+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:03.340804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1447,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.254918+0000","last_change":"2024-09-16T03:50:58.540624+0000","last_active":"2024-09-16T04:21:18.254918+0000","last_peered":"2024-09-16T04:21:18.254918+0000","last_clean":"2024-09-16T04:21:18.254918+0000","last_became_active":"2024-09-16T03:50:58.540379+0000","last_became_peered":"2024-09-16T03:50:58.540379+0000","last_unstale":"2024-09-16T04:21:18.254918+0000","last_undegraded":"2024-09-16T04:21:18.254918+0000","last_fullsized":"2024-09-16T04:21:18.254918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1441,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.082676+0000","last_change":"2024-09-16T03:50:58.480959+0000","last_active":"2024-09-16T04:21:18.082676+0000","last_peered":"2024-09-16T04:21:18.082676+0000","last_clean":"2024-09-16T04:21:18.082676+0000","last_became_active":"2024-09-16T03:50:58.480684+0000","last_became_peered":"2024-09-16T03:50:58.480684+0000","last_unstale":"2024-09-16T04:21:18.082676+0000","last_undegraded":"2024-09-16T04:21:18.082676+0000","last_fullsized":"2024-09-16T04:21:18.082676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:38.866093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1462,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.884138+0000","last_change":"2024-09-16T03:50:58.508324+0000","last_active":"2024-09-16T04:21:17.884138+0000","last_peered":"2024-09-16T04:21:17.884138+0000","last_clean":"2024-09-16T04:21:17.884138+0000","last_became_active":"2024-09-16T03:50:58.507923+0000","last_became_peered":"2024-09-16T03:50:58.507923+0000","last_unstale":"2024-09-16T04:21:17.884138+0000","last_undegraded":"2024-09-16T04:21:17.884138+0000","last_fullsized":"2024-09-16T04:21:17.884138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:35:50.490187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316278+0000","last_change":"2024-09-16T03:50:58.539556+0000","last_active":"2024-09-16T04:21:18.316278+0000","last_peered":"2024-09-16T04:21:18.316278+0000","last_clean":"2024-09-16T04:21:18.316278+0000","last_became_active":"2024-09-16T03:50:58.539098+0000","last_became_peered":"2024-09-16T03:50:58.539098+0000","last_unstale":"2024-09-16T04:21:18.316278+0000","last_undegraded":"2024-09-16T04:21:18.316278+0000","last_fullsized":"2024-09-16T04:21:18.316278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:34.294689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":1618,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316617+0000","last_change":"2024-09-16T03:49:47.840601+0000","last_active":"2024-09-16T04:21:18.316617+0000","last_peered":"2024-09-16T04:21:18.316617+0000","last_clean":"2024-09-16T04:21:18.316617+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T04:21:18.316617+0000","last_undegraded":"2024-09-16T04:21:18.316617+0000","last_fullsized":"2024-09-16T04:21:18.316617+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:46.819202+0000","last_clean_scrub_stamp":"2024-09-16T03:49:46.819202+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:00.553224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.082705+0000","last_change":"2024-09-16T03:49:55.859755+0000","last_active":"2024-09-16T04:21:18.082705+0000","last_peered":"2024-09-16T04:21:18.082705+0000","last_clean":"2024-09-16T04:21:18.082705+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T04:21:18.082705+0000","last_undegraded":"2024-09-16T04:21:18.082705+0000","last_fullsized":"2024-09-16T04:21:18.082705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:50.376298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1442,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.082830+0000","last_change":"2024-09-16T03:50:58.527360+0000","last_active":"2024-09-16T04:21:18.082830+0000","last_peered":"2024-09-16T04:21:18.082830+0000","last_clean":"2024-09-16T04:21:18.082830+0000","last_became_active":"2024-09-16T03:50:58.527023+0000","last_became_peered":"2024-09-16T03:50:58.527023+0000","last_unstale":"2024-09-16T04:21:18.082830+0000","last_undegraded":"2024-09-16T04:21:18.082830+0000","last_fullsized":"2024-09-16T04:21:18.082830+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:42.928632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1466,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316532+0000","last_change":"2024-09-16T03:49:55.856772+0000","last_active":"2024-09-16T04:21:18.316532+0000","last_peered":"2024-09-16T04:21:18.316532+0000","last_clean":"2024-09-16T04:21:18.316532+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T04:21:18.316532+0000","last_undegraded":"2024-09-16T04:21:18.316532+0000","last_fullsized":"2024-09-16T04:21:18.316532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:08.365084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881726+0000","last_change":"2024-09-16T03:50:58.528877+0000","last_active":"2024-09-16T04:21:17.881726+0000","last_peered":"2024-09-16T04:21:17.881726+0000","last_clean":"2024-09-16T04:21:17.881726+0000","last_became_active":"2024-09-16T03:50:58.528663+0000","last_became_peered":"2024-09-16T03:50:58.528663+0000","last_unstale":"2024-09-16T04:21:17.881726+0000","last_undegraded":"2024-09-16T04:21:17.881726+0000","last_fullsized":"2024-09-16T04:21:17.881726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:01.620543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1465,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.880501+0000","last_change":"2024-09-16T03:49:56.127656+0000","last_active":"2024-09-16T04:21:17.880501+0000","last_peered":"2024-09-16T04:21:17.880501+0000","last_clean":"2024-09-16T04:21:17.880501+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T04:21:17.880501+0000","last_undegraded":"2024-09-16T04:21:17.880501+0000","last_fullsized":"2024-09-16T04:21:17.880501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:22.110456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1469,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316109+0000","last_change":"2024-09-16T03:50:58.548783+0000","last_active":"2024-09-16T04:21:18.316109+0000","last_peered":"2024-09-16T04:21:18.316109+0000","last_clean":"2024-09-16T04:21:18.316109+0000","last_became_active":"2024-09-16T03:50:58.545532+0000","last_became_peered":"2024-09-16T03:50:58.545532+0000","last_unstale":"2024-09-16T04:21:18.316109+0000","last_undegraded":"2024-09-16T04:21:18.316109+0000","last_fullsized":"2024-09-16T04:21:18.316109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:46.492270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.081897+0000","last_change":"2024-09-16T03:49:55.860220+0000","last_active":"2024-09-16T04:21:18.081897+0000","last_peered":"2024-09-16T04:21:18.081897+0000","last_clean":"2024-09-16T04:21:18.081897+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T04:21:18.081897+0000","last_undegraded":"2024-09-16T04:21:18.081897+0000","last_fullsized":"2024-09-16T04:21:18.081897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:55.726541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1469,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316999+0000","last_change":"2024-09-16T03:50:58.560987+0000","last_active":"2024-09-16T04:21:18.316999+0000","last_peered":"2024-09-16T04:21:18.316999+0000","last_clean":"2024-09-16T04:21:18.316999+0000","last_became_active":"2024-09-16T03:50:58.555850+0000","last_became_peered":"2024-09-16T03:50:58.555850+0000","last_unstale":"2024-09-16T04:21:18.316999+0000","last_undegraded":"2024-09-16T04:21:18.316999+0000","last_fullsized":"2024-09-16T04:21:18.316999+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:43.936131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1464,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.255068+0000","last_change":"2024-09-16T03:49:55.861804+0000","last_active":"2024-09-16T04:21:18.255068+0000","last_peered":"2024-09-16T04:21:18.255068+0000","last_clean":"2024-09-16T04:21:18.255068+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T04:21:18.255068+0000","last_undegraded":"2024-09-16T04:21:18.255068+0000","last_fullsized":"2024-09-16T04:21:18.255068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881223+0000","last_change":"2024-09-16T03:50:58.561772+0000","last_active":"2024-09-16T04:21:17.881223+0000","last_peered":"2024-09-16T04:21:17.881223+0000","last_clean":"2024-09-16T04:21:17.881223+0000","last_became_active":"2024-09-16T03:50:58.561522+0000","last_became_peered":"2024-09-16T03:50:58.561522+0000","last_unstale":"2024-09-16T04:21:17.881223+0000","last_undegraded":"2024-09-16T04:21:17.881223+0000","last_fullsized":"2024-09-16T04:21:17.881223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:46.038811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1466,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316453+0000","last_change":"2024-09-16T03:49:55.856793+0000","last_active":"2024-09-16T04:21:18.316453+0000","last_peered":"2024-09-16T04:21:18.316453+0000","last_clean":"2024-09-16T04:21:18.316453+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T04:21:18.316453+0000","last_undegraded":"2024-09-16T04:21:18.316453+0000","last_fullsized":"2024-09-16T04:21:18.316453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:05:29.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"24'7","reported_seq":1476,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.884904+0000","last_change":"2024-09-16T03:50:58.476506+0000","last_active":"2024-09-16T04:21:17.884904+0000","last_peered":"2024-09-16T04:21:17.884904+0000","last_clean":"2024-09-16T04:21:17.884904+0000","last_became_active":"2024-09-16T03:50:58.476200+0000","last_became_peered":"2024-09-16T03:50:58.476200+0000","last_unstale":"2024-09-16T04:21:17.884904+0000","last_undegraded":"2024-09-16T04:21:17.884904+0000","last_fullsized":"2024-09-16T04:21:17.884904+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:05.408395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1463,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.254711+0000","last_change":"2024-09-16T03:49:55.861805+0000","last_active":"2024-09-16T04:21:18.254711+0000","last_peered":"2024-09-16T04:21:18.254711+0000","last_clean":"2024-09-16T04:21:18.254711+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T04:21:18.254711+0000","last_undegraded":"2024-09-16T04:21:18.254711+0000","last_fullsized":"2024-09-16T04:21:18.254711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1467,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.317762+0000","last_change":"2024-09-16T03:50:58.522695+0000","last_active":"2024-09-16T04:21:18.317762+0000","last_peered":"2024-09-16T04:21:18.317762+0000","last_clean":"2024-09-16T04:21:18.317762+0000","last_became_active":"2024-09-16T03:50:58.522014+0000","last_became_peered":"2024-09-16T03:50:58.522014+0000","last_unstale":"2024-09-16T04:21:18.317762+0000","last_undegraded":"2024-09-16T04:21:18.317762+0000","last_fullsized":"2024-09-16T04:21:18.317762+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:38.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1473,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.882958+0000","last_change":"2024-09-16T03:49:55.907667+0000","last_active":"2024-09-16T04:21:17.882958+0000","last_peered":"2024-09-16T04:21:17.882958+0000","last_clean":"2024-09-16T04:21:17.882958+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T04:21:17.882958+0000","last_undegraded":"2024-09-16T04:21:17.882958+0000","last_fullsized":"2024-09-16T04:21:17.882958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:49:52.836595+0000","last_clean_scrub_stamp":"2024-09-16T03:49:52.836595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:27.178133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1447,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:21.542910+0000","last_change":"2024-09-16T03:50:58.520798+0000","last_active":"2024-09-16T04:21:21.542910+0000","last_peered":"2024-09-16T04:21:21.542910+0000","last_clean":"2024-09-16T04:21:21.542910+0000","last_became_active":"2024-09-16T03:50:58.519191+0000","last_became_peered":"2024-09-16T03:50:58.519191+0000","last_unstale":"2024-09-16T04:21:21.542910+0000","last_undegraded":"2024-09-16T04:21:21.542910+0000","last_fullsized":"2024-09-16T04:21:21.542910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1465,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.318325+0000","last_change":"2024-09-16T03:50:58.547695+0000","last_active":"2024-09-16T04:21:18.318325+0000","last_peered":"2024-09-16T04:21:18.318325+0000","last_clean":"2024-09-16T04:21:18.318325+0000","last_became_active":"2024-09-16T03:50:58.546981+0000","last_became_peered":"2024-09-16T03:50:58.546981+0000","last_unstale":"2024-09-16T04:21:18.318325+0000","last_undegraded":"2024-09-16T04:21:18.318325+0000","last_fullsized":"2024-09-16T04:21:18.318325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:29.088340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316477+0000","last_change":"2024-09-16T03:50:58.542578+0000","last_active":"2024-09-16T04:21:18.316477+0000","last_peered":"2024-09-16T04:21:18.316477+0000","last_clean":"2024-09-16T04:21:18.316477+0000","last_became_active":"2024-09-16T03:50:58.541780+0000","last_became_peered":"2024-09-16T03:50:58.541780+0000","last_unstale":"2024-09-16T04:21:18.316477+0000","last_undegraded":"2024-09-16T04:21:18.316477+0000","last_fullsized":"2024-09-16T04:21:18.316477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:34.596173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.881253+0000","last_change":"2024-09-16T03:50:58.542966+0000","last_active":"2024-09-16T04:21:17.881253+0000","last_peered":"2024-09-16T04:21:17.881253+0000","last_clean":"2024-09-16T04:21:17.881253+0000","last_became_active":"2024-09-16T03:50:58.542726+0000","last_became_peered":"2024-09-16T03:50:58.542726+0000","last_unstale":"2024-09-16T04:21:17.881253+0000","last_undegraded":"2024-09-16T04:21:17.881253+0000","last_fullsized":"2024-09-16T04:21:17.881253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:11.421727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1470,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316337+0000","last_change":"2024-09-16T03:50:58.558551+0000","last_active":"2024-09-16T04:21:18.316337+0000","last_peered":"2024-09-16T04:21:18.316337+0000","last_clean":"2024-09-16T04:21:18.316337+0000","last_became_active":"2024-09-16T03:50:58.557890+0000","last_became_peered":"2024-09-16T03:50:58.557890+0000","last_unstale":"2024-09-16T04:21:18.316337+0000","last_undegraded":"2024-09-16T04:21:18.316337+0000","last_fullsized":"2024-09-16T04:21:18.316337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:13.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.882902+0000","last_change":"2024-09-16T03:50:58.545310+0000","last_active":"2024-09-16T04:21:17.882902+0000","last_peered":"2024-09-16T04:21:17.882902+0000","last_clean":"2024-09-16T04:21:17.882902+0000","last_became_active":"2024-09-16T03:50:58.544536+0000","last_became_peered":"2024-09-16T03:50:58.544536+0000","last_unstale":"2024-09-16T04:21:17.882902+0000","last_undegraded":"2024-09-16T04:21:17.882902+0000","last_fullsized":"2024-09-16T04:21:17.882902+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:34:15.701626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1461,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.884571+0000","last_change":"2024-09-16T03:50:58.570630+0000","last_active":"2024-09-16T04:21:17.884571+0000","last_peered":"2024-09-16T04:21:17.884571+0000","last_clean":"2024-09-16T04:21:17.884571+0000","last_became_active":"2024-09-16T03:50:58.570205+0000","last_became_peered":"2024-09-16T03:50:58.570205+0000","last_unstale":"2024-09-16T04:21:17.884571+0000","last_undegraded":"2024-09-16T04:21:17.884571+0000","last_fullsized":"2024-09-16T04:21:17.884571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:23.406420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1446,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.255827+0000","last_change":"2024-09-16T03:50:58.545353+0000","last_active":"2024-09-16T04:21:18.255827+0000","last_peered":"2024-09-16T04:21:18.255827+0000","last_clean":"2024-09-16T04:21:18.255827+0000","last_became_active":"2024-09-16T03:50:58.543727+0000","last_became_peered":"2024-09-16T03:50:58.543727+0000","last_unstale":"2024-09-16T04:21:18.255827+0000","last_undegraded":"2024-09-16T04:21:18.255827+0000","last_fullsized":"2024-09-16T04:21:18.255827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1450,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.316283+0000","last_change":"2024-09-16T03:50:58.550926+0000","last_active":"2024-09-16T04:21:18.316283+0000","last_peered":"2024-09-16T04:21:18.316283+0000","last_clean":"2024-09-16T04:21:18.316283+0000","last_became_active":"2024-09-16T03:50:58.549440+0000","last_became_peered":"2024-09-16T03:50:58.549440+0000","last_unstale":"2024-09-16T04:21:18.316283+0000","last_undegraded":"2024-09-16T04:21:18.316283+0000","last_fullsized":"2024-09-16T04:21:18.316283+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:58:40.013005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1440,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.082608+0000","last_change":"2024-09-16T03:50:58.483761+0000","last_active":"2024-09-16T04:21:18.082608+0000","last_peered":"2024-09-16T04:21:18.082608+0000","last_clean":"2024-09-16T04:21:18.082608+0000","last_became_active":"2024-09-16T03:50:58.483098+0000","last_became_peered":"2024-09-16T03:50:58.483098+0000","last_unstale":"2024-09-16T04:21:18.082608+0000","last_undegraded":"2024-09-16T04:21:18.082608+0000","last_fullsized":"2024-09-16T04:21:18.082608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:30.723960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1461,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:17.883961+0000","last_change":"2024-09-16T03:50:58.524457+0000","last_active":"2024-09-16T04:21:17.883961+0000","last_peered":"2024-09-16T04:21:17.883961+0000","last_clean":"2024-09-16T04:21:17.883961+0000","last_became_active":"2024-09-16T03:50:58.524223+0000","last_became_peered":"2024-09-16T03:50:58.524223+0000","last_unstale":"2024-09-16T04:21:17.883961+0000","last_undegraded":"2024-09-16T04:21:17.883961+0000","last_fullsized":"2024-09-16T04:21:17.883961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:41.294641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1446,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:18.254353+0000","last_change":"2024-09-16T03:50:58.520158+0000","last_active":"2024-09-16T04:21:18.254353+0000","last_peered":"2024-09-16T04:21:18.254353+0000","last_clean":"2024-09-16T04:21:18.254353+0000","last_became_active":"2024-09-16T03:50:58.519543+0000","last_became_peered":"2024-09-16T03:50:58.519543+0000","last_unstale":"2024-09-16T04:21:18.254353+0000","last_undegraded":"2024-09-16T04:21:18.254353+0000","last_fullsized":"2024-09-16T04:21:18.254353+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:50:57.417996+0000","last_clean_scrub_stamp":"2024-09-16T03:50:57.417996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67112,"kb_used_data":3808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304728,"statfs":{"total":96636764160,"available":96568041472,"internally_reserved":0,"allocated":3899392,"data_stored":1700524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509822,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67484,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304356,"statfs":{"total":96636764160,"available":96567660544,"internally_reserved":0,"allocated":3903488,"data_stored":1700529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68364,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303476,"statfs":{"total":96636764160,"available":96566759424,"internally_reserved":0,"allocated":4616192,"data_stored":2341360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542527,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79964,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291876,"statfs":{"total":96636764160,"available":96554881024,"internally_reserved":0,"allocated":3903488,"data_stored":1700543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71640,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300200,"statfs":{"total":96636764160,"available":96563404800,"internally_reserved":0,"allocated":4087808,"data_stored":1812437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75616,"kb_used_data":3808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296224,"statfs":{"total":96636764160,"available":96559333376,"internally_reserved":0,"allocated":3899392,"data_stored":1700524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542526,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79776,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292064,"statfs":{"total":96636764160,"available":96555073536,"internally_reserved":0,"allocated":4087808,"data_stored":1812451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76136,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295704,"statfs":{"total":96636764160,"available":96558800896,"internally_reserved":0,"allocated":4431872,"data_stored":2229452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.469 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T03:49:46.819202+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.470 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.471 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T03:49:52.836595+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=49, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.472 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-16T03:50:57.417996+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=50, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=21, tm_sec=19, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:21:24.473 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:21:44.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:21:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 10 monclient: get_monmap_and_config 2024-09-16T04:21:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 10 monclient: build_initial_monmap 2024-09-16T04:21:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 10 monclient: monmap: 2024-09-16T04:21:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:44.584 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:44.583694+0000 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:44.583694+0000 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:44.585 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 1 Processor -- start 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 1 -- start start 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.583+0000 7f87182a5640 10 monclient: init 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87100b5470 addr v1:172.21.15.163:6789/0 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 0x7f871013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-b con 0x7f871013cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:44.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8710143040 addr v1:172.21.15.70:6789/0 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87100ac040 con 0x7f8710143040 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87100ac2b0 con 0x7f87100b5470 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f87100b5f60 con 0x7f871013cbd0 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f87182a5640 10 monclient(hunting): authenticate will time out at 2663.245686s 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 0x7f871013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8716842640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8710143040 0x7f8710143490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:52426/0 (socket says 172.21.15.70:52426) 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8716842640 1 -- 172.21.15.70:0/3348200867 learned_addr learned my addr 172.21.15.70:0/3348200867 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): get_auth_request con 0x7f871013cbd0 auth_method 0 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.584+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 10 monclient(hunting): handle_auth_done global_id 21035 payload 274 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 1 -- 172.21.15.70:0/3348200867 >> v1:172.21.15.163:6789/0 conn(0x7f87100b5470 legacy=0x7f87100b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 1 -- 172.21.15.70:0/3348200867 >> v1:172.21.15.70:6789/0 conn(0x7f8710143040 legacy=0x7f8710143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 10 monclient: _finish_hunting 0 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 1 monclient: found mon.noname-b 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 1 -- 172.21.15.70:0/3348200867 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87100aaef0 con 0x7f871013cbd0 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 1 -- 172.21.15.70:0/3348200867 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825467589 0 0) 0x7f87100ac040 con 0x7f8710143040 2024-09-16T04:21:44.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f8715840640 1 --2- 172.21.15.70:0/3348200867 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 0x7f871013d020 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f870c0097d0 tx=0x7f870c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30b655f962df0f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 1 -- 172.21.15.70:0/3348200867 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c037070 con 0x7f871013cbd0 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: dump: 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.597 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: _finish_auth 0 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: _check_auth_tickets 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 1 -- 172.21.15.70:0/3348200867 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c032440 con 0x7f871013cbd0 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f87182a5640 5 monclient: authenticate success, global_id 21035 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f87182a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 1 -- 172.21.15.70:0/3348200867 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c032810 con 0x7f871013cbd0 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f87182a5640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:44.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f87182a5640 4 set_mon_vals no callback set 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.585+0000 7f871503f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f871503f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f871503f640 10 monclient: dump: 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 10 monclient: shutdown 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 -- 172.21.15.70:0/3348200867 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 msgr2=0x7f871013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 --2- 172.21.15.70:0/3348200867 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 0x7f871013d020 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f870c0097d0 tx=0x7f870c02a250 comp rx=0 tx=0).stop 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 -- 172.21.15.70:0/3348200867 shutdown_connections 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 --2- 172.21.15.70:0/3348200867 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f871013cbd0 0x7f871013d020 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 -- 172.21.15.70:0/3348200867 >> 172.21.15.70:0/3348200867 conn(0x7f87100679c0 msgr2=0x7f8710067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 -- 172.21.15.70:0/3348200867 shutdown_connections 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.586+0000 7f87182a5640 1 -- 172.21.15.70:0/3348200867 wait complete. 2024-09-16T04:21:44.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 10 monclient: build_initial_monmap 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 10 monclient: monmap: 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 librados: starting msgr at 2024-09-16T04:21:44.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 librados: starting objecter 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 Processor -- start 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 -- start start 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 librados: setting wanted keys 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 1 librados: calling monclient init 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.587+0000 7f87182a5640 10 monclient: init 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 0x7f8710146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-b con 0x7f87100b5470 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 0x7f8710145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-a con 0x7f871013cbd0 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): picked mon.noname-c con 0x7f8710143040 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:44.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8710143d20 con 0x7f871013cbd0 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8710143ba0 con 0x7f87100b5470 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f871014e8d0 con 0x7f8710143040 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient(hunting): authenticate will time out at 2663.249184s 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 0x7f8710145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:41538/0 (socket says 172.21.15.70:41538) 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 0x7f8710145270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42086/0 (socket says 172.21.15.70:42086) 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 -- 172.21.15.70:0/2946377531 learned_addr learned my addr 172.21.15.70:0/2946377531 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): get_auth_request con 0x7f8710143040 auth_method 0 2024-09-16T04:21:44.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): get_auth_request con 0x7f871013cbd0 auth_method 0 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8715840640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 0x7f8710146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 10 monclient(hunting): get_auth_request con 0x7f87100b5470 auth_method 0 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient(hunting): handle_auth_done global_id 21038 payload 824 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 -- 172.21.15.70:0/2946377531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 msgr2=0x7f8710146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 0x7f8710146060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 -- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 msgr2=0x7f8710145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 0x7f8710145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient: _finish_hunting 0 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 monclient: found mon.noname-c 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 -- 172.21.15.70:0/2946377531 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8710146e30 con 0x7f8710143040 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716041640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 0x7f8710146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8716842640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f870800c700 tx=0x7f870800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 client.?.objecter ms_handle_connect 0x7f8710143040 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708018070 con 0x7f8710143040 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: dump: 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.606 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: _finish_auth 0 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 5 monclient: authenticate success, global_id 21038 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient: _renew_subs 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870800ce10 con 0x7f8710143040 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8710147110 con 0x7f8710143040 2024-09-16T04:21:44.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870800a880 con 0x7f8710143040 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f8702ffd640 10 monclient: dump: 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.608 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 client.21038.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 monclient: _renew_subs 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.588+0000 7f871483e640 4 set_mon_vals no callback set 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101457b0 con 0x7f8710143040 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 1 librados: init done 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.589+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871013d020 con 0x7f8710143040 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.590+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f870801b070 con 0x7f8710143040 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.590+0000 7f8702ffd640 1 --2- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 0x7f86f40700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8716041640 1 --2- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 0x7f86f40700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8716041640 10 monclient: get_auth_request con 0x7f86f406dc30 auth_method 0 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f870808a6c0 con 0x7f8710143040 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 10 client.21038.objecter ms_dispatch 0x7f8710000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 3 client.21038.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 3 client.21038.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 20 client.21038.objecter dump_active .. 0 homeless 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8702ffd640 10 client.21038.objecter ms_handle_connect 0x7f86f406dc30 2024-09-16T04:21:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.591+0000 7f8716041640 1 --2- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 0x7f86f40700f0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7f8704003eb0 tx=0x7f8704009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.595+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8708058a30 con 0x7f8710143040 2024-09-16T04:21:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.595+0000 7f8702ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.595+0000 7f8702ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:44.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.761+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 --> v2:172.21.15.70:6804/198622594 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8710072230 con 0x7f86f406dc30 2024-09-16T04:21:44.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.764+0000 7f8702ffd640 1 -- 172.21.15.70:0/2946377531 <== mgr.4100 v2:172.21.15.70:6804/198622594 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116265 (secure 0 0 0) 0x7f8704035640 con 0x7f86f406dc30 2024-09-16T04:21:44.766 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-16T04:21:44.767 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-16T04:21:44.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 10 librados: watch_flush enter 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 10 librados: watch_flush exit 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 20 client.21038.objecter shutdown clearing up homeless session... 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 10 client.21038.objecter successfully canceled tick 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 msgr2=0x7f86f40700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 0x7f86f40700f0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7f8704003eb0 tx=0x7f8704009040 comp rx=0 tx=0).stop 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 10 monclient: shutdown 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 msgr2=0x7f871013dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f870800c700 tx=0x7f870800cbc0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 shutdown_connections 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> v2:172.21.15.70:6804/198622594 conn(0x7f86f406dc30 0x7f86f40700f0 unknown :-1 s=CLOSED pgs=4045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f8710143040 0x7f871013dc50 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f871013cbd0 0x7f8710145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 --2- 172.21.15.70:0/2946377531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f87100b5470 0x7f8710146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 >> 172.21.15.70:0/2946377531 conn(0x7f87100679c0 msgr2=0x7f8710067da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 shutdown_connections 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.766+0000 7f87182a5640 1 -- 172.21.15.70:0/2946377531 wait complete. 2024-09-16T04:21:44.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.767+0000 7f87182a5640 1 librados: shutdown 2024-09-16T04:21:44.779 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1706,"stamp":"2024-09-16T04:21:43.178115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":582780,"kb_used_data":32060,"kb_used_omap":12,"kb_used_meta":550579,"kb_avail":754391940,"statfs":{"total":773094113280,"available":772497346560,"internally_reserved":0,"allocated":32829440,"data_stored":14997820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":563793458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002589"},"pg_stats":[{"pgid":"5.15","version":"0'0","reported_seq":1449,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:28.679258+0000","last_change":"2024-09-16T04:21:28.679290+0000","last_active":"2024-09-16T04:21:28.679258+0000","last_peered":"2024-09-16T04:21:28.679258+0000","last_clean":"2024-09-16T04:21:28.679258+0000","last_became_active":"2024-09-16T03:50:58.527591+0000","last_became_peered":"2024-09-16T03:50:58.527591+0000","last_unstale":"2024-09-16T04:21:28.679258+0000","last_undegraded":"2024-09-16T04:21:28.679258+0000","last_fullsized":"2024-09-16T04:21:28.679258+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:28.679163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:28.679163+0000","last_clean_scrub_stamp":"2024-09-16T04:21:28.679163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:46:56.167229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1474,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.346566+0000","last_change":"2024-09-16T04:21:26.346566+0000","last_active":"2024-09-16T04:21:26.346566+0000","last_peered":"2024-09-16T04:21:26.346566+0000","last_clean":"2024-09-16T04:21:26.346566+0000","last_became_active":"2024-09-16T03:50:58.565169+0000","last_became_peered":"2024-09-16T03:50:58.565169+0000","last_unstale":"2024-09-16T04:21:26.346566+0000","last_undegraded":"2024-09-16T04:21:26.346566+0000","last_fullsized":"2024-09-16T04:21:26.346566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.346403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.346403+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.346403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:39:31.442696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1473,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.300183+0000","last_change":"2024-09-16T04:21:23.300220+0000","last_active":"2024-09-16T04:21:23.300183+0000","last_peered":"2024-09-16T04:21:23.300183+0000","last_clean":"2024-09-16T04:21:23.300183+0000","last_became_active":"2024-09-16T03:50:58.520920+0000","last_became_peered":"2024-09-16T03:50:58.520920+0000","last_unstale":"2024-09-16T04:21:23.300183+0000","last_undegraded":"2024-09-16T04:21:23.300183+0000","last_fullsized":"2024-09-16T04:21:23.300183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:23.300094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:23.300094+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.300094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:32:33.089920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1449,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:29.672081+0000","last_change":"2024-09-16T04:21:29.672104+0000","last_active":"2024-09-16T04:21:29.672081+0000","last_peered":"2024-09-16T04:21:29.672081+0000","last_clean":"2024-09-16T04:21:29.672081+0000","last_became_active":"2024-09-16T03:50:58.529616+0000","last_became_peered":"2024-09-16T03:50:58.529616+0000","last_unstale":"2024-09-16T04:21:29.672081+0000","last_undegraded":"2024-09-16T04:21:29.672081+0000","last_fullsized":"2024-09-16T04:21:29.672081+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:29.671963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:29.671963+0000","last_clean_scrub_stamp":"2024-09-16T04:21:29.671963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:08.353230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1461,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:29.363792+0000","last_change":"2024-09-16T04:21:29.363820+0000","last_active":"2024-09-16T04:21:29.363792+0000","last_peered":"2024-09-16T04:21:29.363792+0000","last_clean":"2024-09-16T04:21:29.363792+0000","last_became_active":"2024-09-16T03:50:58.547418+0000","last_became_peered":"2024-09-16T03:50:58.547418+0000","last_unstale":"2024-09-16T04:21:29.363792+0000","last_undegraded":"2024-09-16T04:21:29.363792+0000","last_fullsized":"2024-09-16T04:21:29.363792+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:29.363701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:29.363701+0000","last_clean_scrub_stamp":"2024-09-16T04:21:29.363701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:58:56.817080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:27.550721+0000","last_change":"2024-09-16T04:21:27.550743+0000","last_active":"2024-09-16T04:21:27.550721+0000","last_peered":"2024-09-16T04:21:27.550721+0000","last_clean":"2024-09-16T04:21:27.550721+0000","last_became_active":"2024-09-16T03:50:58.563681+0000","last_became_peered":"2024-09-16T03:50:58.563681+0000","last_unstale":"2024-09-16T04:21:27.550721+0000","last_undegraded":"2024-09-16T04:21:27.550721+0000","last_fullsized":"2024-09-16T04:21:27.550721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:27.550636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:27.550636+0000","last_clean_scrub_stamp":"2024-09-16T04:21:27.550636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:12.257751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.557696+0000","last_change":"2024-09-16T04:21:26.557736+0000","last_active":"2024-09-16T04:21:26.557696+0000","last_peered":"2024-09-16T04:21:26.557696+0000","last_clean":"2024-09-16T04:21:26.557696+0000","last_became_active":"2024-09-16T03:50:58.562917+0000","last_became_peered":"2024-09-16T03:50:58.562917+0000","last_unstale":"2024-09-16T04:21:26.557696+0000","last_undegraded":"2024-09-16T04:21:26.557696+0000","last_fullsized":"2024-09-16T04:21:26.557696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.557594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.557594+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.557594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:43.616610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.598732+0000","last_change":"2024-09-16T04:21:25.598789+0000","last_active":"2024-09-16T04:21:25.598732+0000","last_peered":"2024-09-16T04:21:25.598732+0000","last_clean":"2024-09-16T04:21:25.598732+0000","last_became_active":"2024-09-16T03:50:58.552137+0000","last_became_peered":"2024-09-16T03:50:58.552137+0000","last_unstale":"2024-09-16T04:21:25.598732+0000","last_undegraded":"2024-09-16T04:21:25.598732+0000","last_fullsized":"2024-09-16T04:21:25.598732+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.598633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.598633+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.598633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:40:25.457021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.646817+0000","last_change":"2024-09-16T04:21:24.646837+0000","last_active":"2024-09-16T04:21:24.646817+0000","last_peered":"2024-09-16T04:21:24.646817+0000","last_clean":"2024-09-16T04:21:24.646817+0000","last_became_active":"2024-09-16T03:50:58.545776+0000","last_became_peered":"2024-09-16T03:50:58.545776+0000","last_unstale":"2024-09-16T04:21:24.646817+0000","last_undegraded":"2024-09-16T04:21:24.646817+0000","last_fullsized":"2024-09-16T04:21:24.646817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:24.646751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:24.646751+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.646751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:08.247511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1455,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.224539+0000","last_change":"2024-09-16T04:21:26.224563+0000","last_active":"2024-09-16T04:21:26.224539+0000","last_peered":"2024-09-16T04:21:26.224539+0000","last_clean":"2024-09-16T04:21:26.224539+0000","last_became_active":"2024-09-16T03:50:58.540379+0000","last_became_peered":"2024-09-16T03:50:58.540379+0000","last_unstale":"2024-09-16T04:21:26.224539+0000","last_undegraded":"2024-09-16T04:21:26.224539+0000","last_fullsized":"2024-09-16T04:21:26.224539+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.224444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.224444+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.224444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:09.612940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1450,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.628868+0000","last_change":"2024-09-16T04:21:25.628890+0000","last_active":"2024-09-16T04:21:25.628868+0000","last_peered":"2024-09-16T04:21:25.628868+0000","last_clean":"2024-09-16T04:21:25.628868+0000","last_became_active":"2024-09-16T03:50:58.480684+0000","last_became_peered":"2024-09-16T03:50:58.480684+0000","last_unstale":"2024-09-16T04:21:25.628868+0000","last_undegraded":"2024-09-16T04:21:25.628868+0000","last_fullsized":"2024-09-16T04:21:25.628868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.628779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.628779+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.628779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.785450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1471,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.993490+0000","last_change":"2024-09-16T04:21:23.993517+0000","last_active":"2024-09-16T04:21:23.993490+0000","last_peered":"2024-09-16T04:21:23.993490+0000","last_clean":"2024-09-16T04:21:23.993490+0000","last_became_active":"2024-09-16T03:50:58.507923+0000","last_became_peered":"2024-09-16T03:50:58.507923+0000","last_unstale":"2024-09-16T04:21:23.993490+0000","last_undegraded":"2024-09-16T04:21:23.993490+0000","last_fullsized":"2024-09-16T04:21:23.993490+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:23.993398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:23.993398+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.993398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:18.223939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:27.383461+0000","last_change":"2024-09-16T04:21:27.383521+0000","last_active":"2024-09-16T04:21:27.383461+0000","last_peered":"2024-09-16T04:21:27.383461+0000","last_clean":"2024-09-16T04:21:27.383461+0000","last_became_active":"2024-09-16T03:50:58.539098+0000","last_became_peered":"2024-09-16T03:50:58.539098+0000","last_unstale":"2024-09-16T04:21:27.383461+0000","last_undegraded":"2024-09-16T04:21:27.383461+0000","last_fullsized":"2024-09-16T04:21:27.383461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:27.383365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:27.383365+0000","last_clean_scrub_stamp":"2024-09-16T04:21:27.383365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:44.691151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"754'119","reported_seq":1629,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.320786+0000","last_change":"2024-09-16T04:21:24.320837+0000","last_active":"2024-09-16T04:21:24.320786+0000","last_peered":"2024-09-16T04:21:24.320786+0000","last_clean":"2024-09-16T04:21:24.320786+0000","last_became_active":"2024-09-16T03:49:47.836397+0000","last_became_peered":"2024-09-16T03:49:47.836397+0000","last_unstale":"2024-09-16T04:21:24.320786+0000","last_undegraded":"2024-09-16T04:21:24.320786+0000","last_fullsized":"2024-09-16T04:21:24.320786+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'119","last_scrub_stamp":"2024-09-16T04:21:24.320707+0000","last_deep_scrub":"754'119","last_deep_scrub_stamp":"2024-09-16T04:21:24.320707+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.320707+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:20:29.318034+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"754'2","reported_seq":1469,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.549195+0000","last_change":"2024-09-16T04:21:23.549225+0000","last_active":"2024-09-16T04:21:23.549195+0000","last_peered":"2024-09-16T04:21:23.549195+0000","last_clean":"2024-09-16T04:21:23.549195+0000","last_became_active":"2024-09-16T03:49:53.849215+0000","last_became_peered":"2024-09-16T03:49:53.849215+0000","last_unstale":"2024-09-16T04:21:23.549195+0000","last_undegraded":"2024-09-16T04:21:23.549195+0000","last_fullsized":"2024-09-16T04:21:23.549195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'2","last_scrub_stamp":"2024-09-16T04:21:23.549113+0000","last_deep_scrub":"754'2","last_deep_scrub_stamp":"2024-09-16T04:21:23.549113+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.549113+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:39.945798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00042063300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1451,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:27.689377+0000","last_change":"2024-09-16T04:21:27.689400+0000","last_active":"2024-09-16T04:21:27.689377+0000","last_peered":"2024-09-16T04:21:27.689377+0000","last_clean":"2024-09-16T04:21:27.689377+0000","last_became_active":"2024-09-16T03:50:58.527023+0000","last_became_peered":"2024-09-16T03:50:58.527023+0000","last_unstale":"2024-09-16T04:21:27.689377+0000","last_undegraded":"2024-09-16T04:21:27.689377+0000","last_fullsized":"2024-09-16T04:21:27.689377+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:27.689293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:27.689293+0000","last_clean_scrub_stamp":"2024-09-16T04:21:27.689293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:26:23.236082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1475,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.337448+0000","last_change":"2024-09-16T04:21:25.337480+0000","last_active":"2024-09-16T04:21:25.337448+0000","last_peered":"2024-09-16T04:21:25.337448+0000","last_clean":"2024-09-16T04:21:25.337448+0000","last_became_active":"2024-09-16T03:49:53.848730+0000","last_became_peered":"2024-09-16T03:49:53.848730+0000","last_unstale":"2024-09-16T04:21:25.337448+0000","last_undegraded":"2024-09-16T04:21:25.337448+0000","last_fullsized":"2024-09-16T04:21:25.337448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.337357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.337357+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.337357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:44.401415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038947199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.615545+0000","last_change":"2024-09-16T04:21:23.615565+0000","last_active":"2024-09-16T04:21:23.615545+0000","last_peered":"2024-09-16T04:21:23.615545+0000","last_clean":"2024-09-16T04:21:23.615545+0000","last_became_active":"2024-09-16T03:50:58.528663+0000","last_became_peered":"2024-09-16T03:50:58.528663+0000","last_unstale":"2024-09-16T04:21:23.615545+0000","last_undegraded":"2024-09-16T04:21:23.615545+0000","last_fullsized":"2024-09-16T04:21:23.615545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:23.615470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:23.615470+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.615470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:02:37.856957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1474,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:22.598718+0000","last_change":"2024-09-16T04:21:22.598737+0000","last_active":"2024-09-16T04:21:22.598718+0000","last_peered":"2024-09-16T04:21:22.598718+0000","last_clean":"2024-09-16T04:21:22.598718+0000","last_became_active":"2024-09-16T03:49:54.056558+0000","last_became_peered":"2024-09-16T03:49:54.056558+0000","last_unstale":"2024-09-16T04:21:22.598718+0000","last_undegraded":"2024-09-16T04:21:22.598718+0000","last_fullsized":"2024-09-16T04:21:22.598718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:22.598642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:22.598642+0000","last_clean_scrub_stamp":"2024-09-16T04:21:22.598642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:57.865238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028364800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1478,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:21.900020+0000","last_change":"2024-09-16T04:21:21.900039+0000","last_active":"2024-09-16T04:21:21.900020+0000","last_peered":"2024-09-16T04:21:21.900020+0000","last_clean":"2024-09-16T04:21:21.900020+0000","last_became_active":"2024-09-16T03:50:58.545532+0000","last_became_peered":"2024-09-16T03:50:58.545532+0000","last_unstale":"2024-09-16T04:21:21.900020+0000","last_undegraded":"2024-09-16T04:21:21.900020+0000","last_fullsized":"2024-09-16T04:21:21.900020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:21.899948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:21.899948+0000","last_clean_scrub_stamp":"2024-09-16T04:21:21.899948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:23:29.183245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"754'3","reported_seq":1470,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.583953+0000","last_change":"2024-09-16T04:21:24.583981+0000","last_active":"2024-09-16T04:21:24.583953+0000","last_peered":"2024-09-16T04:21:24.583953+0000","last_clean":"2024-09-16T04:21:24.583953+0000","last_became_active":"2024-09-16T03:49:53.850278+0000","last_became_peered":"2024-09-16T03:49:53.850278+0000","last_unstale":"2024-09-16T04:21:24.583953+0000","last_undegraded":"2024-09-16T04:21:24.583953+0000","last_fullsized":"2024-09-16T04:21:24.583953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'3","last_scrub_stamp":"2024-09-16T04:21:24.583872+0000","last_deep_scrub":"754'3","last_deep_scrub_stamp":"2024-09-16T04:21:24.583872+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.583872+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:11.007063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089687399999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1478,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.868229+0000","last_change":"2024-09-16T04:21:23.868249+0000","last_active":"2024-09-16T04:21:23.868229+0000","last_peered":"2024-09-16T04:21:23.868229+0000","last_clean":"2024-09-16T04:21:23.868229+0000","last_became_active":"2024-09-16T03:50:58.555850+0000","last_became_peered":"2024-09-16T03:50:58.555850+0000","last_unstale":"2024-09-16T04:21:23.868229+0000","last_undegraded":"2024-09-16T04:21:23.868229+0000","last_fullsized":"2024-09-16T04:21:23.868229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:23.868159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:23.868159+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.868159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:54.033085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1472,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.239245+0000","last_change":"2024-09-16T04:21:23.239279+0000","last_active":"2024-09-16T04:21:23.239245+0000","last_peered":"2024-09-16T04:21:23.239245+0000","last_clean":"2024-09-16T04:21:23.239245+0000","last_became_active":"2024-09-16T03:49:54.056552+0000","last_became_peered":"2024-09-16T03:49:54.056552+0000","last_unstale":"2024-09-16T04:21:23.239245+0000","last_undegraded":"2024-09-16T04:21:23.239245+0000","last_fullsized":"2024-09-16T04:21:23.239245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:23.239162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:23.239162+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.239162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:05:59.742513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043158399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1469,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:27.300001+0000","last_change":"2024-09-16T04:21:27.300045+0000","last_active":"2024-09-16T04:21:27.300001+0000","last_peered":"2024-09-16T04:21:27.300001+0000","last_clean":"2024-09-16T04:21:27.300001+0000","last_became_active":"2024-09-16T03:50:58.561522+0000","last_became_peered":"2024-09-16T03:50:58.561522+0000","last_unstale":"2024-09-16T04:21:27.300001+0000","last_undegraded":"2024-09-16T04:21:27.300001+0000","last_fullsized":"2024-09-16T04:21:27.300001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:27.299918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:27.299918+0000","last_clean_scrub_stamp":"2024-09-16T04:21:27.299918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:19:05.487070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1475,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.381449+0000","last_change":"2024-09-16T04:21:26.381472+0000","last_active":"2024-09-16T04:21:26.381449+0000","last_peered":"2024-09-16T04:21:26.381449+0000","last_clean":"2024-09-16T04:21:26.381449+0000","last_became_active":"2024-09-16T03:49:54.056688+0000","last_became_peered":"2024-09-16T03:49:54.056688+0000","last_unstale":"2024-09-16T04:21:26.381449+0000","last_undegraded":"2024-09-16T04:21:26.381449+0000","last_fullsized":"2024-09-16T04:21:26.381449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.381362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.381362+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.381362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:36.691872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039839299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"754'8","reported_seq":1486,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:23.007700+0000","last_change":"2024-09-16T04:21:23.007727+0000","last_active":"2024-09-16T04:21:23.007700+0000","last_peered":"2024-09-16T04:21:23.007700+0000","last_clean":"2024-09-16T04:21:23.007700+0000","last_became_active":"2024-09-16T03:50:58.476200+0000","last_became_peered":"2024-09-16T03:50:58.476200+0000","last_unstale":"2024-09-16T04:21:23.007700+0000","last_undegraded":"2024-09-16T04:21:23.007700+0000","last_fullsized":"2024-09-16T04:21:23.007700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'8","last_scrub_stamp":"2024-09-16T04:21:23.007605+0000","last_deep_scrub":"754'8","last_deep_scrub_stamp":"2024-09-16T04:21:23.007605+0000","last_clean_scrub_stamp":"2024-09-16T04:21:23.007605+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:13.314343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1471,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:22.267157+0000","last_change":"2024-09-16T04:21:22.267184+0000","last_active":"2024-09-16T04:21:22.267157+0000","last_peered":"2024-09-16T04:21:22.267157+0000","last_clean":"2024-09-16T04:21:22.267157+0000","last_became_active":"2024-09-16T03:49:53.849328+0000","last_became_peered":"2024-09-16T03:49:53.849328+0000","last_unstale":"2024-09-16T04:21:22.267157+0000","last_undegraded":"2024-09-16T04:21:22.267157+0000","last_fullsized":"2024-09-16T04:21:22.267157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:22.267089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:22.267089+0000","last_clean_scrub_stamp":"2024-09-16T04:21:22.267089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:29.021494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041647599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1476,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.319160+0000","last_change":"2024-09-16T04:21:24.319178+0000","last_active":"2024-09-16T04:21:24.319160+0000","last_peered":"2024-09-16T04:21:24.319160+0000","last_clean":"2024-09-16T04:21:24.319160+0000","last_became_active":"2024-09-16T03:50:58.522014+0000","last_became_peered":"2024-09-16T03:50:58.522014+0000","last_unstale":"2024-09-16T04:21:24.319160+0000","last_undegraded":"2024-09-16T04:21:24.319160+0000","last_fullsized":"2024-09-16T04:21:24.319160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:24.319097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:24.319097+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.319097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:20.902155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1482,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.237823+0000","last_change":"2024-09-16T04:21:24.237864+0000","last_active":"2024-09-16T04:21:24.237823+0000","last_peered":"2024-09-16T04:21:24.237823+0000","last_clean":"2024-09-16T04:21:24.237823+0000","last_became_active":"2024-09-16T03:49:53.849286+0000","last_became_peered":"2024-09-16T03:49:53.849286+0000","last_unstale":"2024-09-16T04:21:24.237823+0000","last_undegraded":"2024-09-16T04:21:24.237823+0000","last_fullsized":"2024-09-16T04:21:24.237823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:24.237707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:24.237707+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.237707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:13.337234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037747399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1455,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.240610+0000","last_change":"2024-09-16T04:21:25.240637+0000","last_active":"2024-09-16T04:21:25.240610+0000","last_peered":"2024-09-16T04:21:25.240610+0000","last_clean":"2024-09-16T04:21:25.240610+0000","last_became_active":"2024-09-16T03:50:58.519191+0000","last_became_peered":"2024-09-16T03:50:58.519191+0000","last_unstale":"2024-09-16T04:21:25.240610+0000","last_undegraded":"2024-09-16T04:21:25.240610+0000","last_fullsized":"2024-09-16T04:21:25.240610+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.240526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.240526+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.240526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:26.493806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1474,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.310776+0000","last_change":"2024-09-16T04:21:25.310802+0000","last_active":"2024-09-16T04:21:25.310776+0000","last_peered":"2024-09-16T04:21:25.310776+0000","last_clean":"2024-09-16T04:21:25.310776+0000","last_became_active":"2024-09-16T03:50:58.546981+0000","last_became_peered":"2024-09-16T03:50:58.546981+0000","last_unstale":"2024-09-16T04:21:25.310776+0000","last_undegraded":"2024-09-16T04:21:25.310776+0000","last_fullsized":"2024-09-16T04:21:25.310776+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.310695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.310695+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.310695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:45.801080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1460,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:28.353222+0000","last_change":"2024-09-16T04:21:28.353249+0000","last_active":"2024-09-16T04:21:28.353222+0000","last_peered":"2024-09-16T04:21:28.353222+0000","last_clean":"2024-09-16T04:21:28.353222+0000","last_became_active":"2024-09-16T03:50:58.541780+0000","last_became_peered":"2024-09-16T03:50:58.541780+0000","last_unstale":"2024-09-16T04:21:28.353222+0000","last_undegraded":"2024-09-16T04:21:28.353222+0000","last_fullsized":"2024-09-16T04:21:28.353222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:28.353131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:28.353131+0000","last_clean_scrub_stamp":"2024-09-16T04:21:28.353131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:25.823355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1468,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.273426+0000","last_change":"2024-09-16T04:21:25.273453+0000","last_active":"2024-09-16T04:21:25.273426+0000","last_peered":"2024-09-16T04:21:25.273426+0000","last_clean":"2024-09-16T04:21:25.273426+0000","last_became_active":"2024-09-16T03:50:58.542726+0000","last_became_peered":"2024-09-16T03:50:58.542726+0000","last_unstale":"2024-09-16T04:21:25.273426+0000","last_undegraded":"2024-09-16T04:21:25.273426+0000","last_fullsized":"2024-09-16T04:21:25.273426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.273334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.273334+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.273334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:30:41.715229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1479,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:22.916676+0000","last_change":"2024-09-16T04:21:22.916695+0000","last_active":"2024-09-16T04:21:22.916676+0000","last_peered":"2024-09-16T04:21:22.916676+0000","last_clean":"2024-09-16T04:21:22.916676+0000","last_became_active":"2024-09-16T03:50:58.557890+0000","last_became_peered":"2024-09-16T03:50:58.557890+0000","last_unstale":"2024-09-16T04:21:22.916676+0000","last_undegraded":"2024-09-16T04:21:22.916676+0000","last_fullsized":"2024-09-16T04:21:22.916676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:22.916603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:22.916603+0000","last_clean_scrub_stamp":"2024-09-16T04:21:22.916603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:11.435199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1469,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.304075+0000","last_change":"2024-09-16T04:21:26.304097+0000","last_active":"2024-09-16T04:21:26.304075+0000","last_peered":"2024-09-16T04:21:26.304075+0000","last_clean":"2024-09-16T04:21:26.304075+0000","last_became_active":"2024-09-16T03:50:58.544536+0000","last_became_peered":"2024-09-16T03:50:58.544536+0000","last_unstale":"2024-09-16T04:21:26.304075+0000","last_undegraded":"2024-09-16T04:21:26.304075+0000","last_fullsized":"2024-09-16T04:21:26.304075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.303991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.303991+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.303991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:10.311146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1470,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.997186+0000","last_change":"2024-09-16T04:21:25.997211+0000","last_active":"2024-09-16T04:21:25.997186+0000","last_peered":"2024-09-16T04:21:25.997186+0000","last_clean":"2024-09-16T04:21:25.997186+0000","last_became_active":"2024-09-16T03:50:58.570205+0000","last_became_peered":"2024-09-16T03:50:58.570205+0000","last_unstale":"2024-09-16T04:21:25.997186+0000","last_undegraded":"2024-09-16T04:21:25.997186+0000","last_fullsized":"2024-09-16T04:21:25.997186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.997099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.997099+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.997099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:16.532082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1454,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:27.253312+0000","last_change":"2024-09-16T04:21:27.253335+0000","last_active":"2024-09-16T04:21:27.253312+0000","last_peered":"2024-09-16T04:21:27.253312+0000","last_clean":"2024-09-16T04:21:27.253312+0000","last_became_active":"2024-09-16T03:50:58.543727+0000","last_became_peered":"2024-09-16T03:50:58.543727+0000","last_unstale":"2024-09-16T04:21:27.253312+0000","last_undegraded":"2024-09-16T04:21:27.253312+0000","last_fullsized":"2024-09-16T04:21:27.253312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:27.253220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:27.253220+0000","last_clean_scrub_stamp":"2024-09-16T04:21:27.253220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:08.456568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1459,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:30.350885+0000","last_change":"2024-09-16T04:21:30.350922+0000","last_active":"2024-09-16T04:21:30.350885+0000","last_peered":"2024-09-16T04:21:30.350885+0000","last_clean":"2024-09-16T04:21:30.350885+0000","last_became_active":"2024-09-16T03:50:58.549440+0000","last_became_peered":"2024-09-16T03:50:58.549440+0000","last_unstale":"2024-09-16T04:21:30.350885+0000","last_undegraded":"2024-09-16T04:21:30.350885+0000","last_fullsized":"2024-09-16T04:21:30.350885+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:30.350794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:30.350794+0000","last_clean_scrub_stamp":"2024-09-16T04:21:30.350794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:15:59.121923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1449,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:26.669288+0000","last_change":"2024-09-16T04:21:26.669323+0000","last_active":"2024-09-16T04:21:26.669288+0000","last_peered":"2024-09-16T04:21:26.669288+0000","last_clean":"2024-09-16T04:21:26.669288+0000","last_became_active":"2024-09-16T03:50:58.483098+0000","last_became_peered":"2024-09-16T03:50:58.483098+0000","last_unstale":"2024-09-16T04:21:26.669288+0000","last_undegraded":"2024-09-16T04:21:26.669288+0000","last_fullsized":"2024-09-16T04:21:26.669288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:26.669172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:26.669172+0000","last_clean_scrub_stamp":"2024-09-16T04:21:26.669172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:12:28.441213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1470,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:25.029346+0000","last_change":"2024-09-16T04:21:25.029370+0000","last_active":"2024-09-16T04:21:25.029346+0000","last_peered":"2024-09-16T04:21:25.029346+0000","last_clean":"2024-09-16T04:21:25.029346+0000","last_became_active":"2024-09-16T03:50:58.524223+0000","last_became_peered":"2024-09-16T03:50:58.524223+0000","last_unstale":"2024-09-16T04:21:25.029346+0000","last_undegraded":"2024-09-16T04:21:25.029346+0000","last_fullsized":"2024-09-16T04:21:25.029346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:25.029254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:25.029254+0000","last_clean_scrub_stamp":"2024-09-16T04:21:25.029254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:10.967652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1454,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-09-16T04:21:24.241947+0000","last_change":"2024-09-16T04:21:24.241965+0000","last_active":"2024-09-16T04:21:24.241947+0000","last_peered":"2024-09-16T04:21:24.241947+0000","last_clean":"2024-09-16T04:21:24.241947+0000","last_became_active":"2024-09-16T03:50:58.519543+0000","last_became_peered":"2024-09-16T03:50:58.519543+0000","last_unstale":"2024-09-16T04:21:24.241947+0000","last_undegraded":"2024-09-16T04:21:24.241947+0000","last_fullsized":"2024-09-16T04:21:24.241947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:21:24.241880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:21:24.241880+0000","last_clean_scrub_stamp":"2024-09-16T04:21:24.241880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:25:07.270567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66928,"kb_used_data":3808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304912,"statfs":{"total":96636764160,"available":96568229888,"internally_reserved":0,"allocated":3899392,"data_stored":1700524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509826,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66932,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304908,"statfs":{"total":96636764160,"available":96568225792,"internally_reserved":0,"allocated":3903488,"data_stored":1700529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542531,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67628,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304212,"statfs":{"total":96636764160,"available":96567513088,"internally_reserved":0,"allocated":4616192,"data_stored":2341360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542531,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79412,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292428,"statfs":{"total":96636764160,"available":96555446272,"internally_reserved":0,"allocated":3903488,"data_stored":1700543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71272,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300568,"statfs":{"total":96636764160,"available":96563781632,"internally_reserved":0,"allocated":4087808,"data_stored":1812437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75248,"kb_used_data":3808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296592,"statfs":{"total":96636764160,"available":96559710208,"internally_reserved":0,"allocated":3899392,"data_stored":1700524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542530,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79592,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292248,"statfs":{"total":96636764160,"available":96555261952,"internally_reserved":0,"allocated":4087808,"data_stored":1812451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75768,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296072,"statfs":{"total":96636764160,"available":96559177728,"internally_reserved":0,"allocated":4431872,"data_stored":2229452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:21:44.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:21:44.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.878+0000 7f4073795640 10 monclient: get_monmap_and_config 2024-09-16T04:21:44.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.878+0000 7f4073795640 10 monclient: build_initial_monmap 2024-09-16T04:21:44.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.878+0000 7f4073795640 10 monclient: monmap: 2024-09-16T04:21:44.880 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:44.880 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.880 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T04:21:44.878783+0000 2024-09-16T04:21:44.880 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T04:21:44.878783+0000 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.70:6789/0 mon.noname-a-legacy 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c-legacy 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-b 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:3: v2:172.21.15.70:3300/0 mon.noname-a 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr:4: v2:172.21.15.163:3300/0 mon.noname-c 2024-09-16T04:21:44.881 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.878+0000 7f4073795640 1 Processor -- start 2024-09-16T04:21:44.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 -- start start 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient: init 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f406c0b5470 0x7f406c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): picked mon.noname-a con 0x7f406c0b5470 addr v2:172.21.15.70:3300/0 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): picked mon.noname-b con 0x7f406c13cbd0 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 --2- >> v2:172.21.15.163:3300/0 conn(0x7f406c143040 0x7f406c143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): picked mon.noname-c con 0x7f406c143040 addr v2:172.21.15.163:3300/0 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0ac040 con 0x7f406c0b5470 2024-09-16T04:21:44.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 -- --> v2:172.21.15.163:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0ac2b0 con 0x7f406c143040 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f406c0aaef0 con 0x7f406c13cbd0 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4073795640 10 monclient(hunting): authenticate will time out at 2663.540751s 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f406c0b5470 0x7f406c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f406c0b5470 0x7f406c0b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42100/0 (socket says 172.21.15.70:42100) 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:41550/0 (socket says 172.21.15.70:41550) 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 1 -- 172.21.15.70:0/2315140676 learned_addr learned my addr 172.21.15.70:0/2315140676 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:44.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 10 monclient(hunting): get_auth_request con 0x7f406c0b5470 auth_method 0 2024-09-16T04:21:44.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4071531640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 10 monclient(hunting): get_auth_request con 0x7f406c13cbd0 auth_method 0 2024-09-16T04:21:44.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.879+0000 7f4070d30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4070d30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4070d30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4070d30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071d32640 1 --2- 172.21.15.70:0/2315140676 >> v2:172.21.15.163:3300/0 conn(0x7f406c143040 0x7f406c143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071d32640 10 monclient(hunting): get_auth_request con 0x7f406c143040 auth_method 0 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071d32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071d32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient(hunting): handle_auth_done global_id 26856 payload 274 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 -- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 msgr2=0x7f406c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 -- 172.21.15.70:0/2315140676 >> v2:172.21.15.163:3300/0 conn(0x7f406c143040 msgr2=0x7f406c143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 --2- 172.21.15.70:0/2315140676 >> v2:172.21.15.163:3300/0 conn(0x7f406c143040 0x7f406c143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient: _finish_hunting 0 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 monclient: found mon.noname-a 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.70:3300/0 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 -- 172.21.15.70:0/2315140676 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c0aab20 con 0x7f406c0b5470 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4070d30640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.880+0000 7f4071531640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c0b5470 0x7f406c0b5850 secure :-1 s=READY pgs=5770 cs=0 l=1 rev1=1 crypto rx=0x7f406c06dbf0 tx=0x7f405c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a831c4e21625490b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 1 -- 172.21.15.70:0/2315140676 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c037070 con 0x7f406c0b5470 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: dump: 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.895 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 1 -- 172.21.15.70:0/2315140676 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c032440 con 0x7f406c0b5470 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 5 monclient: authenticate success, global_id 26856 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 1 -- 172.21.15.70:0/2315140676 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c032810 con 0x7f406c0b5470 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 4 set_mon_vals no callback set 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:21:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4063fff640 10 monclient: dump: 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 10 monclient: shutdown 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 -- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c0b5470 msgr2=0x7f406c0b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c0b5470 0x7f406c0b5850 secure :-1 s=READY pgs=5770 cs=0 l=1 rev1=1 crypto rx=0x7f406c06dbf0 tx=0x7f405c02a270 comp rx=0 tx=0).stop 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 -- 172.21.15.70:0/2315140676 shutdown_connections 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 --2- 172.21.15.70:0/2315140676 >> v2:172.21.15.163:3300/0 conn(0x7f406c143040 0x7f406c143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c13cbd0 0x7f406c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 --2- 172.21.15.70:0/2315140676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c0b5470 0x7f406c0b5850 unknown :-1 s=CLOSED pgs=5770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 -- 172.21.15.70:0/2315140676 >> 172.21.15.70:0/2315140676 conn(0x7f406c0679c0 msgr2=0x7f406c067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 -- 172.21.15.70:0/2315140676 shutdown_connections 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.881+0000 7f4073795640 1 -- 172.21.15.70:0/2315140676 wait complete. 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 10 monclient: build_initial_monmap 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 10 monclient: monmap: 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-09-16T04:21:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.noname-a 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.noname-b 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.noname-c 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 librados: starting msgr at 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 librados: starting objecter 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 Processor -- start 2024-09-16T04:21:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 -- start start 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 librados: setting wanted keys 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 1 librados: calling monclient init 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.882+0000 7f4073795640 10 monclient: init 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): picked mon.noname-c con 0x7f406c0b5470 addr [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-09-16T04:21:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f406c13cbd0 0x7f406c144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): picked mon.noname-b con 0x7f406c13cbd0 addr [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c143040 0x7f406c1db910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): picked mon.noname-a con 0x7f406c143040 addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f406c143b50 con 0x7f406c143040 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1439d0 con 0x7f406c13cbd0 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): start opening mon connection 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 1 -- --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_getmap magic: 0 -- 0x7f406c143cd0 con 0x7f406c0b5470 2024-09-16T04:21:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4073795640 10 monclient(hunting): authenticate will time out at 2663.544406s 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c143040 0x7f406c1db910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 --2- >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3301/0 says I am v2:172.21.15.70:41558/0 (socket says 172.21.15.70:41558) 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 -- 172.21.15.70:0/1942605954 learned_addr learned my addr 172.21.15.70:0/1942605954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): get_auth_request con 0x7f406c0b5470 auth_method 0 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): get_auth_request con 0x7f406c143040 auth_method 0 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:21:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071d32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient(hunting): handle_auth_done global_id 21044 payload 824 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4070d30640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f406c13cbd0 0x7f406c144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 -- 172.21.15.70:0/1942605954 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f406c13cbd0 msgr2=0x7f406c144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f406c13cbd0 0x7f406c144590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 -- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c143040 msgr2=0x7f406c1db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c143040 0x7f406c1db910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient: _finish_hunting 0 2024-09-16T04:21:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 monclient: found mon.noname-c 2024-09-16T04:21:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 -- 172.21.15.70:0/1942605954 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c0ae340 con 0x7f406c0b5470 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4071531640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f405c02a2b0 tx=0x7f405c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4061ffb640 10 client.?.objecter ms_handle_connect 0x7f406c0b5470 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.883+0000 7f4061ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c037040 con 0x7f406c0b5470 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: dump: 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.905 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c042070 con 0x7f406c0b5470 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 5 monclient: authenticate success, global_id 21044 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 10 monclient: _renew_subs 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c0ae620 con 0x7f406c0b5470 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c030880 con 0x7f406c0b5470 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4061ffb640 10 monclient: dump: 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 33825cf0-3459-43e2-ae47-46b331d213a2 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-09-16T03:48:50.246472+0000 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 3 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-16T04:21:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f40637fe640 4 set_mon_vals no callback set 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 10 client.21044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 10 monclient: _renew_subs 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c0aca40 con 0x7f406c0b5470 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.884+0000 7f4073795640 1 librados: init done 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.885+0000 7f4073795640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.885+0000 7f4073795640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.885+0000 7f4073795640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.885+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c13d020 con 0x7f406c0b5470 2024-09-16T04:21:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.885+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 4 ==== mgrmap(e 5) ==== 90811+0+0 (secure 0 0 0) 0x7f405c030a20 con 0x7f406c0b5470 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 1 --2- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 0x7f40500700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4070d30640 1 --2- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 0x7f40500700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4070d30640 10 monclient: get_auth_request con 0x7f405006dc00 auth_method 0 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 5 ==== osd_map(754..754 src has 704..754) ==== 5257+0+0 (secure 0 0 0) 0x7f405c0ab9c0 con 0x7f406c0b5470 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 10 client.21044.objecter ms_dispatch 0x7f406c000b90 osd_map(754..754 src has 704..754) 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 3 client.21044.objecter handle_osd_map got epochs [754,754] > 0 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 3 client.21044.objecter handle_osd_map decoding full epoch 754 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 20 client.21044.objecter dump_active .. 0 homeless 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4070d30640 1 --2- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 0x7f40500700c0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f4068009560 tx=0x7f4068008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:21:44.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.886+0000 7f4061ffb640 10 client.21044.objecter ms_handle_connect 0x7f405006dc00 2024-09-16T04:21:44.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.890+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f405c0ade20 con 0x7f406c0b5470 2024-09-16T04:21:44.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.890+0000 7f4061ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:44.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:44.890+0000 7f4061ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.057+0000 7f4073795640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:21:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.058+0000 7f4073795640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:21:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.058+0000 7f4073795640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.70:3301/0 2024-09-16T04:21:45.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.058+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 --> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f406c0acc80 con 0x7f406c0b5470 2024-09-16T04:21:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f405c032440 con 0x7f406c0b5470 2024-09-16T04:21:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4061ffb640 10 monclient: handle_config config(1 keys) 2024-09-16T04:21:45.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4061ffb640 1 -- 172.21.15.70:0/1942605954 <== mon.2 v2:172.21.15.70:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7f405c07a380 con 0x7f406c0b5470 2024-09-16T04:21:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4061ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:21:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4061ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:21:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4062ffd640 4 set_mon_vals no callback set 2024-09-16T04:21:45.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.062+0000 7f4062ffd640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T04:21:45.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 10 librados: watch_flush enter 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 10 librados: watch_flush exit 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 20 client.21044.objecter shutdown clearing up homeless session... 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 10 client.21044.objecter successfully canceled tick 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 msgr2=0x7f40500700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 0x7f40500700c0 secure :-1 s=READY pgs=4046 cs=0 l=1 rev1=1 crypto rx=0x7f4068009560 tx=0x7f4068008040 comp rx=0 tx=0).stop 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 10 monclient: shutdown 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 msgr2=0x7f406c147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f405c02a2b0 tx=0x7f405c0046c0 comp rx=0 tx=0).stop 2024-09-16T04:21:45.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 shutdown_connections 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> v2:172.21.15.70:6804/198622594 conn(0x7f405006dc00 0x7f40500700c0 unknown :-1 s=CLOSED pgs=4046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f406c143040 0x7f406c1db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f406c13cbd0 0x7f406c144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 --2- 172.21.15.70:0/1942605954 >> [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] conn(0x7f406c0b5470 0x7f406c147050 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 >> 172.21.15.70:0/1942605954 conn(0x7f406c0679c0 msgr2=0x7f406c0b89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 shutdown_connections 2024-09-16T04:21:45.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 -- 172.21.15.70:0/1942605954 wait complete. 2024-09-16T04:21:45.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-16T04:21:45.065+0000 7f4073795640 1 librados: shutdown 2024-09-16T04:21:45.077 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:21:45.077 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:21:45.078 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:21:45.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.118 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:21:45.119 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:21:45.119 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.161 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:21:45.161 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:21:45.161 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.191 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:21:45.191 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:21:45.191 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.260 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:21:45.260 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:21:45.260 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.294 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:21:45.295 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:21:45.295 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.388 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:21:45.389 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T04:21:45.389 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.434 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T04:21:45.434 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T04:21:45.434 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.474 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T04:21:45.475 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:21:45.475 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T04:21:45.475 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.502 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T04:21:45.502 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:21:45.502 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.543 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:21:45.543 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:21:45.544 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:21:45.544 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.562 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:21:45.562 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:21:45.562 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.618 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:21:45.618 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:21:45.619 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:21:45.637 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:21:45.637 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:21:45.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-09-16T04:21:45.686 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T04:21:45.686 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:21:45.923 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T04:21:45.923 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:21:46.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T04:21:46.017 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:21:46.077 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T04:21:46.078 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:21:46.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T04:21:46.128 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:21:46.343 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T04:21:46.343 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:21:46.435 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T04:21:46.435 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T04:21:46.493 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T04:21:46.494 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T04:21:46.551 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:21:46.553 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/data/mon.a.tgz 2024-09-16T04:21:46.553 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-16T04:21:46.569 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.68acuViBas 2024-09-16T04:21:46.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.68acuViBas 2024-09-16T04:21:46.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.68acuViBas 2024-09-16T04:21:46.805 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.68acuViBas is 363KB 2024-09-16T04:21:46.902 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.68acuViBas 2024-09-16T04:21:46.918 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/data/mon.c.tgz 2024-09-16T04:21:46.918 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-16T04:21:46.973 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.sP41h8oZub 2024-09-16T04:21:46.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.sP41h8oZub 2024-09-16T04:21:47.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.sP41h8oZub 2024-09-16T04:21:47.185 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.sP41h8oZub is 364KB 2024-09-16T04:21:47.280 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.sP41h8oZub 2024-09-16T04:21:47.295 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/data/mon.b.tgz 2024-09-16T04:21:47.295 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-09-16T04:21:47.314 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.A4YNk7h2dl 2024-09-16T04:21:47.314 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.A4YNk7h2dl 2024-09-16T04:21:47.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.A4YNk7h2dl 2024-09-16T04:21:47.543 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.A4YNk7h2dl is 364KB 2024-09-16T04:21:47.642 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.A4YNk7h2dl 2024-09-16T04:21:47.657 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:21:47.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:21:47.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:21:47.737 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:21:47.776 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:21:47.776 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:21:47.778 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi070/crash 2024-09-16T04:21:47.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:21:47.803 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi163/crash 2024-09-16T04:21:47.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:21:47.840 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:21:47.840 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:21:47.847 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:21:47.870 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:21:47.870 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41341.log 2024-09-16T04:21:47.870 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-09-16T04:21:47.870 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.41341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-09-16T04:21:47.870 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37440.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41444.log: /var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38700.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5/var/log/ceph/ceph-client.admin.40946.log: --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38700.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37440.log.gz 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.41341.log.gz 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40183.log: -- /var/log/ceph/ceph-client.admin.40136.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38700.log.gz 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40136.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107650.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107650.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107650.log.gz/var/log/ceph/ceph-client.admin.40308.log: 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38899.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33384.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz/var/log/ceph/ceph-client.admin.41393.log: 2024-09-16T04:21:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40136.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.38703.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: 91.8%/var/log/ceph/ceph-client.admin.38703.log: -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41792.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38703.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108280.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39031.log: gzip/var/log/ceph/ceph-client.admin.41792.log: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108280.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38352.log 2024-09-16T04:21:47.896 INFO:teuthology.orchestra.run.smithi070.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108280.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.40329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108154.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38352.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39720.log: gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108154.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41792.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr: /var/log/ceph/ceph-client.admin.40499.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41340.log: /var/log/ceph/ceph-client.admin.40499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38701.log 91.5% 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107860.log 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38357.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-09-16T04:21:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39672.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107860.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38701.log.gz 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38366.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107860.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40178.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39672.log: /var/log/ceph/ceph-client.admin.38366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32977.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40178.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38366.log.gz 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108406.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.32977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41391.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39693.log 2024-09-16T04:21:47.898 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39672.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107734.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.32977.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108406.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108490.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40952.log: /var/log/ceph/ceph-client.admin.107734.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107734.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40579.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7%/var/log/ceph/ceph-client.admin.108490.log: -- replaced with /var/log/ceph/ceph-client.admin.41391.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40579.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108490.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39278.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-09-16T04:21:47.899 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz 2024-09-16T04:21:47.900 INFO:teuthology.orchestra.run.smithi070.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40178.log.gz 2024-09-16T04:21:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2024-09-16T04:21:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41342.log: -- /var/log/ceph/ceph-client.admin.38185.log 2024-09-16T04:21:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41443.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40579.log.gz 2024-09-16T04:21:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40295.log 2024-09-16T04:21:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38185.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38185.log.gz 2024-09-16T04:21:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107944.log 2024-09-16T04:21:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40295.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:21:47.904 INFO:teuthology.orchestra.run.smithi070.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107944.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41000.log 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107944.log.gz 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40585.log 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41000.log: /var/log/ceph/ceph-client.admin.40585.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37969.log 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41457.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-09-16T04:21:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37969.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37969.log.gz 2024-09-16T04:21:47.908 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40295.log.gz 2024-09-16T04:21:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-09-16T04:21:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2024-09-16T04:21:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40399.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-09-16T04:21:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39848.log 2024-09-16T04:21:47.910 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T04:21:47.910 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33106.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33186.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.admin.33106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33397.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33186.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33106.log.gz 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33397.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33186.log.gz 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33226.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33397.log.gz 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33357.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33146.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33226.log.gz 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33357.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:21:47.911 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33146.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33357.log.gz 2024-09-16T04:21:47.912 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:21:47.912 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.5.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33146.log.gz 2024-09-16T04:21:47.912 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:21:47.912 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33266.log 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40878.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41000.log.gz 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39848.log: 95.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108112.log 2024-09-16T04:21:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40537.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-09-16T04:21:47.915 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.33266.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33266.log.gz 2024-09-16T04:21:47.915 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33308.log 2024-09-16T04:21:47.915 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:21:47.915 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.33308.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33308.log.gz 2024-09-16T04:21:47.915 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:21:47.916 INFO:teuthology.orchestra.run.smithi163.stderr: 92.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:21:47.918 INFO:teuthology.orchestra.run.smithi070.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gzgzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27081.log 2024-09-16T04:21:47.918 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:47.918 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108532.log 2024-09-16T04:21:47.920 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.tmp-client.admin.27081.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27081.log.gz 2024-09-16T04:21:47.920 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108112.log.gz 2024-09-16T04:21:47.920 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38688.log 2024-09-16T04:21:47.921 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41458.log 2024-09-16T04:21:47.921 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38688.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108532.log.gz 2024-09-16T04:21:47.921 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38688.log.gz 2024-09-16T04:21:47.921 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108448.log 2024-09-16T04:21:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-09-16T04:21:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108448.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108448.log.gz 2024-09-16T04:21:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-09-16T04:21:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40116.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41458.log.gz 2024-09-16T04:21:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:21:47.926 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39038.log: 92.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-09-16T04:21:47.926 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39800.log 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108322.log 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108322.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108322.log.gz 2024-09-16T04:21:47.927 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-09-16T04:21:47.928 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40580.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-09-16T04:21:47.929 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107692.log 2024-09-16T04:21:47.929 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41459.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-09-16T04:21:47.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107986.log 2024-09-16T04:21:47.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107692.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107692.log.gz 2024-09-16T04:21:47.931 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39037.log 2024-09-16T04:21:47.932 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107986.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107986.log.gz 2024-09-16T04:21:47.933 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39615.log 2024-09-16T04:21:47.933 INFO:teuthology.orchestra.run.smithi163.stderr: 94.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:21:47.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39037.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39037.log.gz 2024-09-16T04:21:47.934 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:21:47.935 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39615.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2024-09-16T04:21:47.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108196.log 2024-09-16T04:21:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.a.log: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.39693.log.gz 2024-09-16T04:21:47.945 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-09-16T04:21:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108196.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108196.log.gz 2024-09-16T04:21:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-09-16T04:21:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40031.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-09-16T04:21:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-09-16T04:21:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41446.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-09-16T04:21:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:21:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40647.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-09-16T04:21:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108070.log 2024-09-16T04:21:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.2.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39615.log.gz 2024-09-16T04:21:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107818.log 2024-09-16T04:21:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108070.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108070.log.gz 2024-09-16T04:21:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41456.log 2024-09-16T04:21:47.970 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107818.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107818.log.gz 2024-09-16T04:21:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-09-16T04:21:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41456.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41456.log.gz 2024-09-16T04:21:47.971 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33279.log 2024-09-16T04:21:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41793.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-09-16T04:21:47.974 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-09-16T04:21:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.33279.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33279.log.gz 2024-09-16T04:21:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-09-16T04:21:47.977 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39235.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-09-16T04:21:47.977 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T04:21:47.978 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38356.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz 2024-09-16T04:21:47.978 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40545.log 2024-09-16T04:21:47.989 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41073.log 2024-09-16T04:21:47.991 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40545.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40545.log.gz 2024-09-16T04:21:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:21:48.022 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2024-09-16T04:21:48.029 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-16T04:21:48.035 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38364.log: 88.2% 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:21:48.036 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-09-16T04:21:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40863.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-09-16T04:21:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108238.log 2024-09-16T04:21:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41794.log: 90.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2024-09-16T04:21:48.050 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-09-16T04:21:48.051 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108238.log: -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-16T04:21:48.059 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-09-16T04:21:48.063 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39035.log: 87.7% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108238.log.gz 2024-09-16T04:21:48.072 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-09-16T04:21:48.072 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38361.log: -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2024-09-16T04:21:48.073 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-09-16T04:21:48.079 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39039.log 2024-09-16T04:21:48.088 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41455.log: 90.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2024-09-16T04:21:48.094 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39039.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-09-16T04:21:48.097 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40669.log 2024-09-16T04:21:48.102 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39867.log: -- replaced with /var/log/ceph/ceph-client.admin.39039.log.gz 2024-09-16T04:21:48.110 INFO:teuthology.orchestra.run.smithi070.stderr: 93.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-09-16T04:21:48.113 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40669.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2024-09-16T04:21:48.123 INFO:teuthology.orchestra.run.smithi070.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.41073.log.gz 2024-09-16T04:21:48.133 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40669.log.gz 2024-09-16T04:21:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107902.log 2024-09-16T04:21:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40383.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-09-16T04:21:48.145 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-09-16T04:21:48.159 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107902.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.107902.log.gz 2024-09-16T04:21:48.159 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41392.log 2024-09-16T04:21:48.160 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38367.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-09-16T04:21:48.177 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-09-16T04:21:48.179 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41392.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41392.log.gz 2024-09-16T04:21:48.198 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-09-16T04:21:48.199 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38686.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-09-16T04:21:48.210 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2024-09-16T04:21:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41445.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-09-16T04:21:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107776.log 2024-09-16T04:21:48.232 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38101.log 2024-09-16T04:21:48.233 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.107776.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107776.log.gz 2024-09-16T04:21:48.234 INFO:teuthology.orchestra.run.smithi070.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2024-09-16T04:21:48.241 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38025.log 2024-09-16T04:21:48.245 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38101.log.gz 2024-09-16T04:21:48.250 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-09-16T04:21:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-09-16T04:21:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40556.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38025.log.gz 2024-09-16T04:21:48.277 INFO:teuthology.orchestra.run.smithi070.stderr: 91.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2024-09-16T04:21:48.277 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40921.log: -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-09-16T04:21:48.288 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38360.log 2024-09-16T04:21:48.289 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39034.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2024-09-16T04:21:48.302 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-09-16T04:21:48.303 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38360.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38360.log.gz 2024-09-16T04:21:48.311 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40997.log 2024-09-16T04:21:48.320 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40343.log: 92.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39872.log 2024-09-16T04:21:48.327 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40997.log: -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-09-16T04:21:48.329 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38143.log 2024-09-16T04:21:48.329 INFO:teuthology.orchestra.run.smithi070.stderr: 91.8%/var/log/ceph/ceph-client.admin.39872.log: -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-09-16T04:21:48.345 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108028.log 2024-09-16T04:21:48.346 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38143.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38143.log.gz 2024-09-16T04:21:48.354 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38702.log 2024-09-16T04:21:48.360 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108028.log: 87.1% 92.1% -- replaced with /var/log/ceph/ceph-client.admin.40997.log.gz 2024-09-16T04:21:48.367 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108028.log.gz 2024-09-16T04:21:48.374 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-09-16T04:21:48.375 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38702.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38702.log.gz 2024-09-16T04:21:48.387 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108572.log 2024-09-16T04:21:48.396 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-16T04:21:48.405 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.108572.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-09-16T04:21:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.41442.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.108572.log.gz 2024-09-16T04:21:48.416 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-09-16T04:21:48.421 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-16T04:21:48.422 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-09-16T04:21:48.426 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39320.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-09-16T04:21:48.429 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/ceph-client.admin.39872.log.gz 2024-09-16T04:21:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39637.log 2024-09-16T04:21:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39915.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-09-16T04:21:48.449 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:21:48.468 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40584.log 2024-09-16T04:21:48.478 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-09-16T04:21:48.478 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.40584.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.39848.log.gz 2024-09-16T04:21:48.488 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108364.log 2024-09-16T04:21:48.492 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.39946.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40584.log.gz 2024-09-16T04:21:48.503 INFO:teuthology.orchestra.run.smithi070.stderr: 90.4%/var/log/ceph/ceph-client.admin.108364.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108364.log.gz 2024-09-16T04:21:48.503 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz 2024-09-16T04:21:48.526 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39637.log.gz 2024-09-16T04:21:48.546 INFO:teuthology.orchestra.run.smithi070.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T04:21:49.053 INFO:teuthology.orchestra.run.smithi163.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:21:49.569 INFO:teuthology.orchestra.run.smithi070.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39800.log.gz 2024-09-16T04:21:49.651 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:21:51.064 INFO:teuthology.orchestra.run.smithi070.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:21:54.194 INFO:teuthology.orchestra.run.smithi163.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T04:21:54.508 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:21:54.679 INFO:teuthology.orchestra.run.smithi163.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:21:54.791 INFO:teuthology.orchestra.run.smithi163.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T04:21:54.837 INFO:teuthology.orchestra.run.smithi163.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:21:54.839 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-09-16T04:21:54.839 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m6.942s 2024-09-16T04:21:54.840 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m27.073s 2024-09-16T04:21:54.840 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m0.954s 2024-09-16T04:21:54.975 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:21:55.685 INFO:teuthology.orchestra.run.smithi070.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:21:55.808 INFO:teuthology.orchestra.run.smithi070.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:21:55.810 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-16T04:21:55.811 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m7.951s 2024-09-16T04:21:55.811 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m35.501s 2024-09-16T04:21:55.811 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m1.239s 2024-09-16T04:21:55.811 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:21:55.811 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi070/log 2024-09-16T04:21:55.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:22:00.781 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi163/log 2024-09-16T04:22:00.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:22:05.257 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:22:05.265 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:22:05.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:22:05.268 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:22:05.354 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:22:05.362 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:22:05.362 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:22:05.366 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:22:05.380 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:22:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 57 -333us[ -324us] +/- 51ms 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 +1153us[+1153us] +/- 55ms 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 120 -895us[ -889us] +/- 90ms 2024-09-16T04:22:05.385 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 7 377 120 -314us[ -255us] +/- 50ms 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 123 +1149us[+1208us] +/- 54ms 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -960us[ -960us] +/- 64ms 2024-09-16T04:22:05.387 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:22:05.388 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:22:05.396 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:22:05.397 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:22:05.407 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:22:05.494 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:22:05.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:22:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:22:05.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:22:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:22:05.618 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:22:05.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:22:05.636 INFO:teuthology.task.internal:Duration was 2361.418121 seconds 2024-09-16T04:22:05.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:22:05.645 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:22:05.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:22:05.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:22:05.685 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:22:05.698 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:22:05.938 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:22:05.938 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-09-16T04:22:05.938 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:22:05.999 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-09-16T04:22:05.999 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:22:06.027 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:22:06.028 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:22:06.041 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:22:06.093 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:22:06.093 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:22:06.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:22:06.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:22:06.245 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:22:06.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:22:06.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:22:06.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:22:06.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:22:06.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:22:06.333 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-09-16T04:22:06.359 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-09-16T04:22:06.380 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:22:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:22:06.419 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:22:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:22:06.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:22:06.446 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:22:06.448 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi070 2024-09-16T04:22:06.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:22:06.522 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905920/remote/smithi163 2024-09-16T04:22:06.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:22:06.560 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:22:06.561 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:22:06.564 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:22:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:22:06.629 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:22:06.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:22:06.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:22:06.639 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:22:06.642 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:22:06.654 INFO:teuthology.orchestra.run.smithi070.stdout: 265613 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:22 /home/ubuntu/cephtest 2024-09-16T04:22:06.664 INFO:teuthology.orchestra.run.smithi163.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:22 /home/ubuntu/cephtest 2024-09-16T04:22:06.666 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:22:06.675 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:22:06.705 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} duration: 2361.4181208610535 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:22:06.705 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:22:06.779 INFO:teuthology.run:pass